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SECTION  1 


INTRODUCTION 


During  the  source  selection  for  a  software  intensive  system,  an  offeror  is  usually  evaluated  on 
his  software  engineering  approach  for  managing  and  developing  the  software  for  the  subject  system. 
Areas  of  evaluation  include  the  offeror’s  methodologies,  tool  sets,  software  development  plan  (SDP) 
aiKl  staffing.  While  evaluation  of  an  offeror’s  software  engineering  ^roach  during  source  selection 
gives  insight  into  how  the  offeror  intends  to  implement  the  software  for  the  system,  the  evaluation  is 
limited  because  it  cannot  give  insight  into  the  offeror’s  own  expertise  with  the  selected  methodology. 
Frequently,  the  Government  has  assessed  an  offeror's  software  engineering  approach  during  source 
selection  as  adequate  only  to  discover  once  on  contract  that  the  offeror  is  not  well  versed  in  the 
proposed  methodology  and  tool  set  or  that  the  offeror  does  rjot  follow  the  SDP.  As  a  result  of  the 
offeror’s  lack  of  expertise  in  the  selected  software  development  approach  or  failure  to  follow  a  firm 
plan,  significant  cost  and  schedule  slips  are  often  encountered  during  the  software  development  phase. 

In  an  attempt  to  limit  further  occurrences  of  this  situation,  the  Electronic  Systems  Division 
(ESD)  of  the  Air  Force  Systems  Command  (AFSQ  determined  the  need  for  a  method  to  be  used  during 
source  selection  for  evaluating  not  only  an  offeror’s  software  development  plan  but  also  the  offeror’s 
expertise  in  the  proposed  software  developmertt  aj^roach.  The  nml  for  this  method  was  perceived  as 
even  greater  within  the  next  few  years  due  to  the  recent  Department  of  Defense  (DOD)  directives  that 
mandated  the  use  of  Ada  as  an  implementation  language.  It  was  feared  that  proposals  would  be 
submitted  by  offerors  who  were  not  well  trained  in  Ada  as  a  software  engineering  methodology.  To 
resolve  this  situation,  therefore,  ESD  and  MITRE  conceived  the  idea  of  a  source  selection  software 
engineering  exercise  (SEE).  As  conceived,  the  purpose  of  the  exercise  was  to  measure  the  degree  of 
risk  associated  with  the  offeror's  software  development  approach  by  testing  the  offeror’s  proposed 
methodology,  as  demonstrated  through  the  offeror's  actual  implementation  of  a  small  exercise  system, 
and  the  offeror’s  ability  to  organize  a  SEE  team  knowledgeable  in  the  proposed  software  engineering 
approach  and  Ada. 

The  Command  Center  Pitxxssing  and  Display  System-Replacement  (CCPDS-R)  program  was 
the  first  of  five  ESD  programs  to  date  to  use  a  SEE  during  source  selection.  ESD,  with  technical 
support  from  The  MITRE  Corporation,  tailored  the  SEE  corxiept  for  CCPDS-R,  determined  the 
approach  for  incorporating  the  SEE  into  the  source  selection  process,  and  drafted  the  actual  exercise 
specification  which  would  serve  as  the  basis  for  the  CCPDS-R  SEE.  Prior  to  actually  using  the  SEE 
during  the  CCPDS-R  source  selection,  the  Goverrunent  determined  that  the  best  way  to  finalize  the  SEE 
concept,  specification  and  evaluation  approach  was  for  MITRE  to  implement  the  exercise  itself.  In  that 
way,  the  Government  would  best  be  able  to  assess  the  feasibility  of  the  SEE,  itKluding  the  scope  of  the 
SEE  and  the  time  that  would  be  made  available  to  the  offerors  for  conducting  the  SEE;  to  ensure  that  the 
exercise  specification  which  would  be  given  to  the  offerors  was  well  written  and  sufficiently 
challenging;  and  to  identify  meaningful  criteria  that  would  be  used  to  evaluate  the  offeror's  SEE  results. 

This  report  provides  a  history  of  the  activities  atxl  decisions  made  in  defining  and  carrying  out 
the  SEE  for  the  CCPDS-R  full  scale  development^xtxluction  (FSD/P)  phase  source  selecticm,  together 
with  a  rationale  for  those  activities  aixi  decisions,  and  a  discussion  of  the  Government's  experiences 
using  the  SEE  on  CCPDS-R.  In  particular,  it  describes  MITRE's  dry  run  of  the  SEE  as  part  of  the 
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source  selection  preparation  effort,  and  identifies  lessons  learned  prior  to  the  start  of  the  source 
selection.  It  also  describes  the  actual  execution  of  the  SEE  during  the  CCPDS-R  source  selection  and 
the  lessons  learned  during  that  period.  Appendices  to  this  report  contain  the  CCPDS-R  SEE  exercise 
specification  and  ground  rules  provided  to  the  CCPDS-R  FSD/P  offerors,  as  well  as  the  SEE 
information  included  in  the  CCT*DS-R  FSD/P  request  for  proposal  (RFP)  package. 


1.1  CCPDS-R  BACKGROUND 

1.1.1  System  Description 

CCPDS-R  will  replace  the  current  Command  Center  Processing  and  Display  System  (CCPDS) 
and  the  missile  warning  fiinction  of  the  NORAD  Computer  System.  The  current  CCPDS  is  located  at 
four  command  centers:  Headquarters,  Strategic  Air  Command  (SAC),  Cheyenne  Mountain  Air  Force 
Base  (CMAFB),  National  Military  Command  Center  (NMCC),  and  the  Alternate  National  Military 
Command  Center  (ANMCQ.  CCPDS  is  dedicated  to  the  receipt,  processing  and  display  of  ballistic 
missile  tactical  warning  and  attack  assessment  (TW/AA)  information.  In  addition,  CCPDS  performs 
associated  command  center  unique  functions  for  use  by  the  national  commarxl  authorities,  chairman  of 
the  Joint  Chiefs  of  Staff,  Commander-in-Chief,  SAC,  and  Commaixler-in-Chief,  North  American 
Aerospace  Defense  Commarxl  in  making  decisions  related  to  the  execution  of  the  single  integrated 
operation  plan,  force/command,  control  and  communications  survival,  arxl  the  use  of  strategic  reserves 
during  all  phases  of  nuclear  engagement. 

As  defined  by  the  new  integrated  tactical  warning  and  attadc  assessment  (ITW&A)  architecture, 
the  CCPDS-R  will  consist  of  four  subsystems.  These  are  the  CMAFB  subsystem,  the  Offutt 
Processing  and  Correlation  Center  (OPCQ  subsystem,  the  SAC  subsystem,  arxl  the  processing  and 
display  subsystem  (PDS).  The  PDS  is  to  te  located  at  the  CMAFB,  NMCC,  ANMCC,  OPCC,  and 
SAC. 


The  CMAFB  arxl  OPCC  subsystems,  referred  to  as  the  coiiunon  subsystems,  will  have 
identical  hardware  arxl  software.  They  will  interface  to  all  ballistic  missile  sensors  via  survivable  and 
non-survivable  media,  process  the  information  received  from  those  sensors,  generate  displays  for  local 
consoles,  integrate  the  missile  warning  information  with  other  manually  entered  data  on  air,  space,  and 
intelligence,  arxl  have  the  capability  fot  distributing  this  correlated  information  to  other  command 
centers  and  subscribers.  The  two  common  subsystems  will  process  the  same  sensor  information  and 
serve  as  mutual  backups  in  case  of  failure  of  critical  components.  Both  subsystems  will  be  able  to 
distribute  correlated  ITW&A  data  to  subscribers  at  a  given  time. 

The  SAC  subsystem  will  be  physically  separate  from  the  OPCC  subsystem  arxl  will  be  solely 
devoted  to  the  support  of  the  SAC  force  management  and  force  survival  missions.  It  will  receive  data 
from  either  the  OrcC  or  CMAFB  common  subsystems,  from  PDS,  and  from  command-unique 
interfaces.  It  will  process  the  data  and  generate  displays  for  consoles  located  at  the  SAC  command 
center  and  other  locations  at  Offutt  Air  Force  Base. 

The  PDS  subsystem  will  be  capable  of  receiving  and  displaying  correlated  ITW&A  information 
from  the  common  subsystem,  direct  badlistic  missile  sensor  data,  and  communication  systems  status 
from  the  Survivable  Communications  Integration  System  (SCIS).  It  will  be  the  primary  system  for 
presentation  of  ITW&A  information  at  the  NMCC,  ANMCC,  and  SAC. 
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1.1.2  Program  Description 

The  CCPDS-R  acquisiticHi  program  consists  of  two  phases:  a  concept  definition/design  (CD/D) 
phase  and  a  full-scale  development^roduction  pdiase.  The  CCPDS-R  FSD/P  effort  is  primarily  a 
software  intensive  effort  using  Depi^ent  of  Defense  Standard  (DOD-STD)  2167  [1],  Ada  as  the 
design  language,  and,  unless  a  waiver  is  granted,  Ada  as  the  implementation  language.  The  CCP£>S-R 
FSD/P  effort  thus  requires  that  contractors  be  prepared  to  design  and  develop  a  real-time  systan  in  Ada 
using  modem  software  engineering  practices.  The  CCPDS-R  FSD/P  contract  was  award^  in  June 
1987  to  TRW. 

The  CCPDS-R  CD/D  phase,  a  year-long  effort  that  coiKluded  in  August  1986,  was  primarily  a 
study  effort  The  CD/D  contractors  were  TRW  and  Ford  Aerospace  and  Conununicatiorrs  Corporation, 
both  of  which  were  expected  to  bid  on  the  FSD/P  contract  During  die  CD/D  phase,  the  contractors 
were  required  to  submit  draft  SDPs.  They  were  also  required  to  perform  several  Ada-related  activities 
to  analyze  the  feasibility  of  using  Ada  as  ^e  CCPDS-R  implementation  language  and  to  demonstrate  the 
contractor’s  capability  to  design  and  develop  a  system  in  Ada,  should  the  contractor  propose  to  use  Ada 
as  the  implementation  language.  The  specific  Ada-related  activities  iiKluded: 

a.  Assess  the  feasibility  of  using  Ada  as  the  CCPDS-R  implementation  language,  and  evaluate 
current  Ada  programming  support  envirwiments  for  their  suitability  of  meeting  CCPDS-R 
requirements 

b .  Devise  a  plan  for  efficient  transition  to  Ada,  if  it  is  determined  that  the  use  of  Ada  is  not 
feasible  on  this  program  at  the  present  time 

c.  Define  and  condua  a  demonstration  that  shows  the  contractor’s  readiness  to  use  Ada,  if  it 
is  determined  that  the  use  of  Ada  is  feasible  on  this  program  now 

d .  Provide  the  rationale  for  choosing  the  Ada-based  design  language  (AIX)  and  present  an 
example  of  the  ADL. 

Despite  the  above  aaivities,  the  Government  determined  that  the  CD/D  contractors  had  not  yet 
adequately  demonstrated  their  ability  to  design  and  implonent  a  real-time  system  in  Ada  using  modem 
software  engineering  practices.  In  general,  the  contractors  had  not  demcmstrated  an  erxl-to-end 
application  of  their  methodologies  and  Ada;  they  had  only  demonstrated  the  features  of  their  tool  sets. 
Therefore,  since  software  development  and  Ada  constitute  major  risks  on  CCPDS-R,  the  Government 
recognized  the  need  for  an  additional  method  to  better  evaluate  the  software  engineering  and  Ada 
capabilities  of  these  two  contractors  and,  mote  importantly,  of  any  other  offerors  who  might  submit 
proposals  for  the  FSD/P  phase.  To  that  end,  the  Government  developed  the  software  engineering 
exercise  as  part  of  the  FSD/P  source  selection  process. 


1.2  OVERVIEW 
1.2.1  SEE  Objectives 

The  Government  viewed  the  SEE  as  a  practical  way  to  assess  each  offeror’s  software 
engineering  capability  prior  to  FSD/P  contract  award.  The  SEE,  which  consists  of  a  small  system  to  be 


3 


designed  by  each  of  the  offerors,  was  intended  to  measure  the  degree  of  risk  associated  with  the 
offeror’s  software  development  methodology,  as  documented  in  the  offeror’s  software  development 
plan.  It  focused  specific^y  on  the  offeror’s  software  develc^ment  methodology,  as  demonstrated  by 
the  actual  t^Iication  of  the  methodology  to  the  exercise  system,  and  on  the  offeror’s  ability  to  organize 
a  team  for  the  SEE,  fully  knowledgeable  in  the  pix^x>sed  methodology  and  Ada. 

Based  on  the  offerors’  performance  on  the  SEE,  the  Government  expected  that  it  would  be 
better  able  to  evaluate  the  offerors’  probability  of  success.  In  particular,  if  an  offeror  failed  the  exercise, 
it  would  be  assessed  that  the  offeror  had  low  probability  of  implementing  CCPDS-R  within  the 
proposed  cost  and  schedule.  If  an  offeror  successfully  completed  the  exercise,  it  would  not  guarantee 
that  the  offeror  would  be  able  to  complete  CCPDS-R  successfully;  however,  it  would  provide  some 
level  of  confidence  in  the  offeror’s  ability  to  implement  CCPDS-R.  In  either  case,  it  would  provide 
early  identification  of  problem  areas  in  the  offeror's  software  a{^roach,  thereby  enabling  the 
Government  to  concentrate  on  these  areas  immediately  at  the  start  of  the  FSD/P  frfiase,  should  the 
offeror  be  awarded  the  contract 

The  Government  did  not  inteixl  to  evaluate  every  aspect  of  software  development  via  the  SEE. 
In  particular,  the  Government  did  not  plan  to  evaluate  those  areas  that  either  would  not  scale  up  to  a 
large  software  development  effort  or  would  not  provide  meaningful  or  discriminating  source  selection 
information.  As  eventually  defined,  the  SEE  was  intended  to  evaluate  the  requirements  analysis  and 
design  methodologies,  the  actual  SEE  design,  and  the  team  expertise  for  the  exercise.  The  SEE  was  rrot 
intended  to  evaluate  coding,  testing,  integration,  productivity,  quality  assurance,  configuration 
management,  software  metrics,  full  compliance  with  DOD-STD-2167,  schedule,  and  management  of 
subcontractors.  The  Government  elected  to  evaluate  the  offerors’  proposals  in  these  areas  by  following 
the  traditional  source  selection  evaluation  approach. 

1.2.2  CCPDS-R  SEE  System  Description 

For  the  CCPDS-R  SEE  to  be  a  meaningful  measure  of  an  offeror’s  ability  to  design  and 
develop  a  real-time  system  like  CCPDS-R,  the  Government  felt  that  the  SEE  system  would  have  to 
require  analysis  of  quantitative  performance  requirements  and  concurrent  processing,  like  CCPDS-R,  it 
would  have  to  be  relevant  to  the  CCPDS-R  mission,  and  it  would  have  to  be  of  suitable  size  and 
complexity  so  that  it  could  be  done  in  a  reasonably  short  period  of  time.  The  system  devised  for  the 
SEE  consists  of  a  missile-warning  scenario  generator  and  simulator.  The  exercise  system  allows  the 
user  to  create  and  edit  scenarios  consisting  of  missile-warning  events  (where  an  event  is  a  missile 
launch  or  nuclear  detonation)  and  to  run  in  real-time  a  missile  warning  simulation  controlled  by  a 
selected  scenario.  The  exercise  system  also  allows  the  user  to  be  able  to  run  a  particular  scenario 
simulation  while  editing  that  same  scenario  file.  This  requirement  forces  the  offerors  to  address  real¬ 
time,  concurrent  operations  comparable  to  those  found  in  CCPDS-R.  Appendix  A  contains  the 
CCPDS-R  SEE  system  specification  as  provided  to  thf  offerors. 


1.3  SCOPE 

This  report  summarizes  the  Government’s  efforts  on  the  software  engineering  exercise  both  in 
preparation  for  and  during  the  CCPDS-R  FSD/P  source  selection.  Sections  2  through  4  address 
Government  SEE  efforts  prior  to  the  start  of  the  CCPDS-R  FSD/P  source  selection.  In  particular, 
section  2  addresses  MITRE’s  own  approach  for  dry  running  the  SEE,  section  3  describe  the  plans 
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devised  by  the  Government  for  evaluating  the  offerors’  SEE  results  during  source  selection,  and  section 
4  summarizes  the  lessons  learned  from  the  MITRE  dry  run  of  the  SEE.  Sections  5  through  7  describe 
the  actual  execution  of  the  SEE  during  the  CCPDS-R  FSD/P  source  selection.  Section  5  summarizes 
the  actual  source  selection  conduct  of  the  SEE,  including  the  process  of  issuing  the  SEE  to  the  offerors, 
the  SEE  products  received  from  the  offerors,  tmd  the  Government's  approach  to  evaluation  of  the 
offeror's  SEE  products.  Section  6  describes  the  lessons  learned  from  administering  the  SEE  during 
source  selection,  arxl  section  7  summarizes  the  offerors’  feedback  concerning  the  use  of  the  SEE. 
Finally,  section  8  provides  an  overall  summary  of  conclusions  arxl  reaxnmendadons  reached  both  prior 
to  conducting  the  SEE  during  the  CCPDS-R  FSD/P  source  selection  and  as  a  result  of  using  the  SEE  in 
the  CCPDS-R  FSD/P  source  selectioa 


5 


SECTION  2 


MITRE  DRY  RUN 


Once  the  Government  devised  the  SEE  concept  and  comi^eted  the  SEE  requirements  definition 
and  preliminary  SEE  system  specihcadon  (also  referr^  to  as  the  exercise  specification),  but  prior  to 
giving  the  SEE  to  the  offerors,  MITRE  assembled  a  team  to  dry  run  the  SEE.  The  primary  objectives  of 
the  dry  run  were  to  generate  a  clearly  defined  SEE  system  specification,  to  develop  the  ground  rules  for 
the  offerors  to  follow  when  conducting  the  SEE,  to  identify  a  set  of  discriminating  SEE  source  selection 
evaluation  criteria,  and  to  assess  whether  the  SEE  could  reasonably  be  done  in  the  time  allotted  to  the 
offerors.  The  secondary  objectives  of  the  effort  were  to  further  educate  CCPDS-R  staff  in  requirements 
analysis  and  design  methodologies,  ADL,  and  Ada,  and  to  gain  familiarity  with  DOD-STD-2167,  a  new 
software  development  staitdard  for  E)OD  acquisitions. 

This  section  describes  the  MITRE  dry  run  of  the  SEE.  In  particular,  it  discusses  the  makeup  of 
the  MITRE  SEE  team;  the  tools,  techniques,  and  methodologies  selected  for  carrying  out  the 
implementation,  and  the  approaches  taken  for  educating  team  members  in  them;  the  schedule;  the 
activities  that  occurred  during  requirements  analysis;  the  activities  that  occurred  during  the  design  phase; 
an  overview  of  the  resulting  SEE  design;  and  the  source  selection  documentation  that  was  product  for 
the  SEE  using  the  results  of  the  dry  run. 


2,1  SEE  TEAM 


The  MITRE  SEE  team  consisted  of  eight  people  with  assigned  roles.  The  particular  roles, 
along  with  the  planned  percentage  of  total  time  to  be  devoted  to  the  SEE,  are  as  shown  below. 

Role  Number  of  Individuals  Percentage  of  Tune 


User/"Govemment"  Representative  1  30 

Software  Development  Manager  1  30 

Technical  Lead  2  80, 50 

Ada  Consultant  I  5 

Designer  2  80, 60 

Designer/Recorder  1  80 


All  team  members  had  in  common  a  computer  software  background  arxl  knowledge  of  either 
PASCAL  or  similar  higher  order  languages.  Only  two  team  members  could  be  considered  both 
software  engineering/Ada  experts  with  extensive  experience.  Two  other  team  members  had 
considerable  Ada  experience,  while  the  remaining  team  members  had  little  or  no  actual  Ada  experience. 
At  the  start  of  the  dry  run,  the  team  had  no  software  development  methodology  or  tool  set  in  place. 
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2.2  SELECTED  TOOLS  AND  METHODOLOGIES 


To  overcome  its  lack  of  a  predetermined  software  development  methodology  and  environment, 
the  SEE  team  selected  a  number  of  tools  and  methodologies  for  dy  running  the  SEE,  and  then 
undertook  efforts  to  become  educated  in  them.  The  particular  tools  and  methodologies  selected  were  as 
follows; 

a.  The  team  elected  to  develop  the  exercise  system  in  accordance  with  DOD-STD-2 1 67,  as 
tailored  for  the  CCPDS-R  program  [2],  and  using  Ada  as  the  design  language,  since  the 
offerors  would  be  required  to  do  this. 

b .  For  a  desigrt  methodology,  the  team  selected  object-oriented  design  (OOD)  as  defined  by 
Grady  Booch  in  "Software  Engineering  with  Ada"  [3].  Booch's  version  of  OOD  was 
selected  because  it  was  one  of  the  better  kix)wn  and  documented  methodologies,  several  of 
the  team  members  were  acquainted  with  it,  and  it  was  expected  that  potential  CCPDS-R 
FSD/P  offerors  might  propose  a  similar  methodology. 

c.  For  a  software  development  environment,  the  team  chose  the  VAX/VMS  Ada  environment 
since  it  was  readily  accessible  via  the  MITRE  Bedford  Computer  Center,  most  team 
members  were  familiar  with  it,  and  it  provided  sufficient  capabilities  to  meet  the  demands 
of  the  exercise. 

d .  As  a  graphical  design  representation  technique,  the  team  selected  Buhr  diagrams  as  defined 
in  "System  Design  With  Ada"  [4]  because  ^  technique  is  designed  for  use  with  Ada,  it  is 
compatible  with  Booch's  OOD,  and  it  provided  a  more  extensive  mapping  from  Ada  and 
design  constructs  than  did  Booch's  notation. 

e.  For  the  Ada-based  design  language,  the  team  chose  a  draft  ADL  standard  that  had  been 
developed  for  another  ESD  project.  A  number  of  team  members  were  acquainted  with  it. 

f.  The  team  did  not  select  any  particular  methodology  for  requirements  analysis,  primarily 
because  the  team  initially  felt  that  the  exercise  was  relatively  smaU,  all  members  understood 
the  requirements  clearly,  and  no  data  flow/data  dictionary  tools  were  readily  available. 

To  become  educated  in  all  of  these  selected  tools  and  procedures,  the  team  studied  numerous 
articles,  participated  in  group  discussions,  and  conducted  demonstrations  under  the  tutelage  of  the 
technical  leads  and  consultant  The  total  time  allocated  throughout  the  effort  for  education  in  the  tools 
and  methodologies  was  minimal,  estimated  at  approximately  five  days  distributed  over  a  2-week  period. 


2.3  SCHEDULE 

Prior  to  commencing  the  actual  design  and  development  of  the  SEE,  the  team  technical  leads 
developed  a  schedule  and  work  plan  for  the  effort  Ftgure  1  depicts  this  initial  projected  schedule.  This 
schedule,  though  longer  than  that  anticipated  for  the  CCPDS-R  offerors,  was  considered  justifiaMe 
since  the  team  required  training  in  the  methodology,  which  the  offerors  should  rat  the  team  members 
were  rat  dedicated  full  time,  as  the  offerors'  members  were  expected  to  be;  atxl  the  team  needed  to 
prepare  additional  documentation  rat  required  of  the  offerors.  As  figure  1  reveals,  the  projected  effort 
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Figure  1.  Projected  MITRE  SEE  Dry-Run  Schedule 
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would  extend  over  ^IXP.  month  period,  with  1  week  allocated  for  requirements  analysis,  3  1/2  weeks 
for  design,  1  1/2  weeks  for  coding  of  selected  portions,  and  2  1/2  weeks  for  developing  the  source 
selection  documentation  (e.g.,  final  exercise  specification,  evaluation  criteria,  etc.)-  This  initial 
schedule  represented  an  accelerated  effort  based  on  an  anticipated  IS  My  1986  release  of  the  CXTPDS-R 
RFP  package.  The  actual  schedule  followed  for  the  SEE  dry  run,  however,  turned  out  to  be 
considerably  longer.  Figure  2  depicts  the  actual  MITRE  SEE  dry  run  schedule.  The  primary  reasons 
why  the  team  deviated  from  the  original  schedule  were  that  team  members  were  unaUe  to  devote  as 
much  time  as  originally  planned,  particular  efforts,  such  as  requirements  analysis,  took  much  longer 
than  estimated  (see  section  4.1.2),  and  uruelated  delays  occurred  in  the  CCPDS-R  RFP  release  which 
obviated  the  need  for  the  original,  accelerated  schedule. 


2.4  REQUIREMENTS  ANALYSIS  ACTIVITIES 

The  team  commenced  its  dry  run  of  the  SEE  by  conducting  an  analysis  of  the  SEE  specification 
requirements.  Input  to  this  requirements  analysis  effort  was  the  draft  SEE  system  specification 
described  in  section  1.2.2.  The  team  assumed  at  the  start  of  the  requirements  analysis  phase  that  the 
draft  exercise  specification  was  essentially  free  of  major  ambiguities  and  inconsistencies.  This 
assumption  was  based  on  a  quick  reading  of  the  draft  specification  and  the  feeling  of  the  team  that  such 
a  short  specification  probably  did  not  have  any  serious  problems  in  it.  The  team's  main  objectives  for 
this  phase  were  to  define  a  software  architecture  that  clearly  identified  the  computer  software 
configuration  items  (CSCIs)  for  the  exercise  system,  to  create  adequately  detailed  software  requirements 
specifications  (SRSs)  that  provided  the  technically  important  portions  of  the  DOD-STD-2167  data  item 
(DI)  description  (DID),  such  as  the  definition  of  inter-CSCI  interfaces,  and  to  identify  any  ambiguities 
remaining  in  the  exercise  specification. 

The  team's  first  step  in  the  requirements  analysis  effort  was  the  development  of  an  overall 
software  architecture  for  the  exercise  system.  The  software  architecture  which  the  team  developed 
consisted  of  four  CSCIs:  two  application-level  CSCIs,  the  missile  warning  simulator  (MWS)  and  the 
scenario  generator  (SG);  a  user-system  interface  (USI)  CSQ;  and  a  file  manager  (FM)  CSCI.  Figure  3 
uses  Buhr  notation  to  depict  the  major  components  of  this  architecture  and  the  control  and  data  flows 
among  them.  As  figure  3  reveals,  there  are  two  major,  independent  control  threads  that  tie  the  system 
together  the  first  passes  from  USI  through  SG  to  FM,  and  the  second  passes  from  USI  through  the 
MWS  to  FM.  In  the  absence  of  other  guidelines,  the  team  decided  to  allocate  particular  requirements  to 
each  CSCI  so  as  to  reflect  most  accurately  and  straightforwardly  the  requirements  breakdown  in  the 
draft  exercise  specification.  Also,  the  team  decided  to  decompose  the  system  into  these  four  distinct 
CSCIs  rather  than  one  CSCI  with  four  computer  software  components  (CSCs)  for  two  primary 
reasons:  first,  the  team  wanted  to  make  the  design  non-trivial  so  that  the  team  would  be  forced  to  deal 
immediately  with  issues  of  interface  definition  aixl  perfonnance  allocation;  aixl  second,  the  team  wished 
to  view  the  exercise  specification  as  if  it  was  a  real  specification  that  required  a  high  level  decomposition 
and  the  creation  of  at  least  two  SRSs. 

Upon  development  of  the  overall  software  architecture  for  the  exercise  system,  the  SEE  team 
carried  out  a  number  of  other  activities  and  generated  specific  products.  The  specific  activities 
conducted  and  products  generated  during  the  requirements  an^ysis  phase  included 
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Figures.  MITRE'S  Overall  SEE  Architecture 


a.  DocumentaticMi  for  most  SRS  sections  for  all  four  CSCIs.  Sections  of  the  SRSs  not 
prepared  included  adaptation  requirements,  qualification  requirements,  and  quality  factors. 
These  sections  were  not  generat^  either  because  the  team  did  not  have  sufficient  time  to 
prepare  them,  the  team  did  not  expect  the  offerors  to  complete  these  sections  in  their  allotted 
time  to  conduct  the  SEE,  or  the  sections  did  not  provide  any  elaboration  of  requirements 
contained  in  the  exercise  specificatitm. 

b.  A  partial  allocation  of  timing  budgets  to  CSCIs.  A  complete  allocation  was  not  performed 
because  there  was  insufficient  time  to  finish  this  task  properly  and  because  the  team  did  not 
have  control  over  the  target  executitm  environment  (a  time-shared  VAX). 

c.  Interface  specifications  for  each  CSCI-to-CSQ  interface. 

d .  Oata  flow  diagrams  for  selected  fiiiKtitMis  and  a  global  data  diaionary. 

e.  A  "mock"  software  specification  review  (SSR)  in  accordance  with  DOD-STT)-2167. 

f.  A  revised  draft  of  the  system  specification  that  reflected  the  discussions  held  during  the 
definition  of  the  CSCIs  and  their  interfaces  (see  section  2.7.1). 

Although  the  team  did  not  apply  a  formal  requirements  analysis  methodology,  the  use  of  a  data 
dictionary  and  data  flow  diagrams  was  sufflcient  for  the  team  to  complete  the  other  efforts  identified 
above,  to  develop  the  overall  software  architecture,  and  hence,  to  adiieve  all  of  the  requirements 
analysis  phase  objectives.  The  team  did  feel  that  a  more  comprehensive  exercise  than  the  one  defined 
would  have  forced  the  use  of  a  fornial  methodology. 


2.5  DESIGN  ACTIVITIES 

The  SEE  team  started  the  design  phase  of  the  dry  run  upon  completion  of  the  mock  SSR  and 
review  of  the  draft  SRS  documents.  The  team's  objective  for  this  phase  was  to  raise  as  many  Ada- 
related  methodology  and  design  issues  as  possible.  It  was  not  the  team's  objective  to  develop  a 
complete,  fully  documented  design.  The  team  selected  two  of  the  CSCIs,  scenario  generator  and  file 
manager,  for  which  to  conduct  preliminary  design.  The  team  picked  these  two  CSQs  for  four  reasons: 
first,  these  CSQs  shared  a  non-trivial  interface  that  required  the  joint,  consistent  specification  of  data 
elements,  control  flow,  and  timing  budgets;  second,  these  CSQs  fonned  a  portion  of  one  of  the  two 
major  independent  control  threads  in  the  exercise  system;  third,  the  correa  operation  of  the  file  manager 
required  that  the  preliminary  design  show  evidence  that  certain  concurrent  readAvrite  issues  had  been 
resolved;  and  fourth,  of  the  SRS  documents  prepared  by  the  team,  the  SRSs  for  these  CSQs  were  the 
most  detailed. 

As  stated  in  section  2.2,  the  team  carried  out  preliminary  design  for  the  two  selected  CSQs 
using  Booch's  OOD  methodology.  In  addition,  the  team  developed  Buhr  diagrams  and  ADL  for  the  SG 
and  FM  CSQs.  Figure  4  is  a  sample  of  one  of  tiie  Buhr  diagrams  produced  for  a  file  manager 
function,  R^rieve.Event  The  team  did  not  develop  formal  software  top-level  design  documents 
(STLDDs)  for  these  CSQs,  due  to  lack  of  time;  however,  the  team  made  most  of  the  technical  decisions 
needed  for  these  documents  and  presented  the  results  at  a  mock  preliminary  design  review  (PDR).  The 
team  did  no  further  design  work  following  the  mock  PDR,  the  reasons  being  diat  team  members  could 
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FILE  MANAGEMENT  SERVICES 


Figure  4.  Samite  Buhr  Diagram:  RETRIEVE_EVENT 


no  longer  devote  laige  amounts  of  time  to  the  diy  nm  and  members  felt  that  no  additional  discriminating 
design  issues  would  be  raised  by  continued  design  decompositioa 

At  the  conclusion  of  the  design  phase,  the  team  identified  a  number  of  Ada-related  methodology 
and  design  issues  which  were  encoijntered  during  the  dry  run.  The  most  prominent  methodology  issue 
was  the  transition  to  preliminary  design  from  requirements  analysis,  following  the  guidelines  in 
Booch's  OOD  methodology,  and  in  particular,  the  introduction  of  ADL.  The  most  important  design 
issue  was  related  to  the  Ada  tasking  model:  the  prioritization  of  tasks  and  the  avoidance  of  deadlock, 
race  conditions,  and  task  starvation.  A  second  impmtant  issue  concented  the  treatment  of  system 
initialization  and  termination,  artd  their  interrelation  with  the  Ada  elaboration  rules.  With  the 
identification  of  these  and  other  issues  (see  section  4  for  a  discussion  of  these  issues),  the  team  satisfied 
its  design-phase  dry-run  objective. 


2.6  RESULTING  SEE  DESIGN 

At  the  completion  of  the  design-phase  dry  mn,  the  design  for  the  SEE  system  which  emerged 
consisted  of  a  menu-driven  system  containing  fourCSCIs,  each  running  asynchronously.  In  the 
design,  USI  is  an  Ada  task  which  generates  the  menus  used  to  solicit  input  commands  from  the  user, 
validates  all  user  inputs;  forwards  valid  scenario  generator  and  missile  warning  simulator  commatvis  for 
processing  to  SG  and  MWS,  respectively;  accepts  data  from  SG  and  MWS,  and  generates  appropriate 
menus  to  solicit  input  conunands  or  missile  warning  disfdays.  SG  is  an  Ada  task  that  performs  scenario 
generation  processing,  allowing  the  user  to  create,  edit,  delete  and  save  scenario  files  consisting  of 
missile  launch  and  nuclear  detonation  message  events.  MWS  is  an  Ada  task  that  performs  the 
simulation  processing  for  a  particular  scenario  file;  that  is  to  say,  it  performs  processing  on  the  event 
messages  in  the  simulation  scenario,  calculates  missile  warning  dismay  information  elements  based  on 
the  contents  of  the  event  messages,  and  makes  the  information  elements  available  for  display  by  USI. 
Finally,  FM  is  an  Ada  task  that  provides  a  common  set  of  mechanisms  for  SG  and  MWS  to  access  a 
centralized  database  of  scenario  files  and  to  prioritize  requests  by  SG  and  MWS  for  access  to  the 
scenario  files. 


2.7  SEE  DOCUMENTATION 

As  stated  in  section  2,  two  of  the  primary  objectives  of  the  MTTRE  SEE  dry  nm  were  to 
generate  a  clearly  defined  SEE  system  specification  and  to  develop  the  ground  rules  for  the  offerors  to 
follow  when  conducting  the  SEE.  The  Government  considered  these  products  essential  to  scope  the 
exercise,  to  achieve  a  meaningful  exercise,  and  to  ensure  commonality  among  ofieror  tq>proaches  and 
efforts  (e.g.,  what  hardware  could  be  used  arxl  what  products  were  to  be  generated  by  the  offerors  as 
part  of  the  exercise),  thereby  enaUing  the  Government  to  evaluate  the  offerors'  SEE  results  in  an 
objective  and  consistent  maimer.  The  actual  SEE  system  specification  and  ground  rules,  or  detailed 
instructions  for  the  offeror,  which  w  rc  generated  upon  completion  of  the  dry  run  of  the  SEE  are 
contained  in  appendix  A.  A  description  of  these  documents  and  their  derivation  is  contained  below. 

2.7.1  SEE  Specification 

MITRE  commeiKed  the  SEE  dry  run  using  a  draft  SEE  system  specification.  At  the  completion 
of  the  dry  run,  when  determining  whether  to  modify  this  specification  in  certain  areas,  an  issue  was 
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raised  as  to  the  appiopriate  level  of  detail  for  the  specification;  some  team  members  wanted  very  detailed 
requirements  in  the  specification,  and  some  felt  that  very  detailed  requirements  were  inappropriate  since 
they  implied  design.  The  resolution  for  this  dilemma  was  to  incorporate  the  very  detailed  requirements 
into  the  specification  only  if  diey  were  necessary  to  bound  the  scope  of  the  exercise;  otherwise,  the 
detailed  requirements  were  omitted  and  left  as  a  design  issue  for  the  offerors.  Thus,  the  major 
modifications  the  team  made  to  the  draft  specification  as  a  result  of  the  dry  r\m  concerned  the  areas  of 
hardware,  growth  and  flexibility,  and  performance  requirements. 

2.7.1. 1  Hardware 

The  draft  SEE  system  specification  stated  only  that  no  special  hardware  was  needed  for  the 
exercise  system.  The  team  modified  the  SEE  specification,  however,  to  require  that  no  special  graphics 
hardware  or  capat»lities  be  used  and  that  the  user  interface  be  designed  to  operate  on  a  single  dumb 
terminal  with  keyboard  entry  device.  The  team  made  these  changes  to  ensure  a  level  of  commonality 
among  the  offerors'  designs  and  to  preclude  the  offerors  from  focusing  their  efforts  on  sophisticated 
graphics  capabilities  at  the  expense  of  addressing  key  software  design  issues. 

2.7.1.2  Growth  and  Flexibility 

Initially,  the  SEE  system  specification  had  rto  requirements  for  growth  artd  flexibility  of  the 
exercise  system.  Since  gro>vth  and  flexibility  ate  key  requirements  of  the  CCPDS-R  system,  the  team 
elected  to  add  requirements  to  the  SEE  specification  in  ttese  areas  so  that  the  offerors  could  be  evaluated 
on  their  approaches  for  handling  growth  and  flexibility.  In  particular,  the  team  added  both  a  general  and 
a  detailed  set  of  growth  and  flexibility  requirements.  The  general  requirement  specified  that  the  design 
be  modular  to  facilitate  changes  in  software  components  which  are  needed  to  accommodate  future 
changes  in  operational  requirements.  The  detail^  requirements  specified  that  the  system  include  the 
capability  for  the  user  to  query  an  individual  scenario  file  based  on  a  fixed  set  of  criteria,  arxl  that  the 
system  be  flexible  enough  to  allow  as  future  growth  the  capability  for  the  user  to  query  across  multiple 
scenario  files  for  this  same  set  of  fixed  criteria. 

2.7.1.3  Performance  Requirements 

The  draft  exercise  specification  contained  no  performance  requirements.  Again,  since 
performance  is  a  key  lequiiement  of  the  CCPDS-R  system,  the  team  decided  to  add  detailed 
performance  requirements  and  load  cotKlitions  to  the  SEE  specification  so  that  the  offerors  could  be 
evaluated  on  their  approaches  for  addressing  performance  issues.  As  part  of  this  approach,  the  team 
included  in  the  SEE  specification  some  exact  performance  requirements  from  the  CCPDS-R  system 
specification  [2].  The  team  also  left  some  of  the  SEE  performance  requirements  ambiguous.  For 
example,  some  of  the  SEE  requirements  were  unclear  as  to  the  end  points  for  measuring  performance. 
This  approach  of  leaving  ambiguous  requirements  in  the  specificatitxi  provided  the  Government  the 
opportunity  to  evaluate  the  offerors'  methodologies  for  their  ability  to  handle  one  of  the  key  functions  of 
requirements  analysis,  namely,  the  detection  and  resolution  of  specification  ambiguities. 

2.7.2  Instructions  for  the  Offeror 

In  addition  to  the  SEE  system  specification,  at  the  comf^etion  of  the  SEE  dry  run,  the  S^  team 
generated  a  detailed  set  of  offeror  instructions.  The  major  ground  rules  included  in  the  detailed 
instructions  for  the  offeror  (see  appendix  A)  pertained  to  the  exercise  scope  and  duration.  Government 
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interaction,  the  offeror’s  methodologies  and  tools,  team  composition,  and  deliverable  products  and  their 
formats. 


Exercise  Scope  and  Duration 

Based  on  the  results  of  the  dry  run  of  the  SEE,  the  team  reached  the  following  conclusions 
regarding  the  scope  and  duration  of  the  exercise: 

a.  A  complete  implementation  of  the  exercise  system,  from  requirements  analysis 
through  coding  and  testing,  could  not  be  done  within  the  time  period  allocated  during  the 
CCPDS-R  source  selectioa 

b.  Coding  and  testing  of  the  exercise  system  would  not  provide  significant  discriminating 
information  regarding  one's  ability  to  design  and  implement  a  real-time  system  in  Ada. 

Use  of  Ada  as  a  design  language  during  the  preliminary  and  detailed  design  phases 
provided  sufficient  information  to  assess  one's  ability  in  these  areas. 

c.  Allowing  the  exercise  period  to  exceed  four  weeks  was  not  considered  beneficial. 
Comparable  to  a  college  "take  home"  examination  which  has  a  point  at  which  no  further 
improvement  in  quality  is  achieved,  it  was  determined  that  no  further  discriminatory 
information  could  be  obtained  by  allowing  the  exercise  period  to  extend  beyond  four  weeks 
to  six  or  eight  weeks,  for  example.  In  fact,  having  the  exercise  go  beyond  four  weeks 
could  be  detrimental  since  it  could  result  in  an  overload  of  SEE  material  for  the  Government 
to  evaluate. 

Given  the  above  conclusions,  the  team  specified  in  the  detailed  instructions  for  the  offerors  that 
the  offerors  develop  a  complete  software  architecture  for  the  exercise  system;  coixluct  requiremerits 
analysis  and  preliminary  design  for  two  or  more  onnponents  of  that  architecture,  with  the  components 
to  be  selected  by  the  offeror,  and  conduct  detailed  design  for  one  or  more  components  of  the 
architecture,  again  with  the  ccxnponents  selected  by  the  offeror.  This  would  provide  the  Government 
with  sample  products  from  each  major  software  development  i^iase  with  minimal  burden  on  the  offeror. 
Also,  the  team  specified  that  the  exercise  duratitm,  from  offeror  receipt  of  the  SEE  specification  and 
ground  rules  until  delivery  of  the  completed  products,  be  limited  to  3  1/2  weeks. 

2.7 .2.2  Government  Interaction 

The  MITRE  dry  run  of  the  SEE  was  conducted  in  accordance  with  DOD-STD-2167,  as  tailored 
for  CCPDS-R  [2].  As  such,  it  included  some  of  the  typical  reviews  held  during  the  software 
development  effort,  such  as  the  software  specification  review  and  preliminary  design  review.  During 
these  reviews,  participating  personnel  assumed  the  roles  of  Government  acquisition  agencies. 
Government  using  agetKies,  and  contractors.  Conducting  these  reviews  provided  the  "contractors"  the 
opportunity  to  submit  questions  to  the  "Government"  to  obtain  clarification  of  requirements,  resolution 
of  specification  ambiguities,  and  design  verificatioa  During  the  conduct  of  these  reviews,  it  became 
evident  that  CCPDS-R  offerors  might  develop  similar  questions  during  their  implementation  of  the  SEE 
which  would  require  resolution.  In  the  interest  of  fairness,  it  was  considered  undesirable  to  have  any 
interaction  between  the  Government  arxl  the  offerors  during  the  exercise  period,  since  cme  offeror  might 
inadvertently  be  given  more  information  or  direction  than  another.  Therefore,  in  the  recommended 
instructions  for  the  offeror,  the  team  explicitly  stated  that  there  would  be  no  interaction  between  the 
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offerors  and  the  Government  during  the  offerors'  execution  of  the  exercise.  Should  the  offerors  have 
any  questions  on  the  exercise,  the  offerors  were  instructed  to  identify  appropriate  assumptioits.  to 
document  those  assumptions,  and  to  proceed  with  the  exercise  based  on  those  assumptions. 

2.7.2.3  Methodologies  and  Tools 

During  the  dry  run  of  the  SEE,  the  observation  was  made  that  while  a  particular  methodology 
may  be  considered  complete  and  satisfactory  in  theory,  it  may  turn  out  to  require  modification  once  it  is 
actuaUy  used  on  a  real  apfdication.  This  was  considered  true  for  the  object-oriented  design 
methodology  used  by  the  SEE  team  (see  section  4.2).  The  instructions  for  the  offeror  specified  that  all 
offerors  must  follow  their  proposed  requirements  ai^ysis  and  design  methodologies  as  documented  in 
the  SDPs  submitted  with  t^  CCPDS-R  technical  proposal;  however,  the  offerors  were  also  allowed  to 
submit  with  their  delivered  SEE  products  changes  to  their  SDPs  which  provided  further  concise, 
technical  details  regarding  the  methodologies  used  during  the  SEE  requirements  analysis  and  design 
phases.  These  changes  would  be  considered  part  of  the  offeror’s  technical  proposal  and  subject  to 
Government  evaluation. 

The  observation  was  also  made  that  familiarity  with  the  selected  tool  set  was  essential  in  order 
to  promote  ease  of  design  and  development  The  fact  that  a  number  of  the  team  members  were  not  well 
versed  in  the  selected  VAX  Ada  environment  and  tool  set  slowed  progress.  However,  to  require  that 
the  CCPDS-R  offerors  use  the  actual  tool  sets  proposed  CCPDS-R  did  not  tqjpear  suitable  since  the 
offerors  might  not  have  all  the  tools  in  house.  (It  was  not  considered  proper  for  the  Government  to 
mandate  that  offerors  expend  funds  to  obtain  these  tools  for  the  SEE.)  Consequently,  in  the 
recommended  instructions  for  the  offeror,  the  team  specified  only  that  the  offerors  use  the  tool  set 
proposed  for  CCPDS-R  to  the  maximum  extent  practical,  as  this  would  be  viewed  more  favorably  by 
the  Government 

2.7.2.4  Team  Composition 

During  the  course  of  the  SEE  dry  nin,  it  became  evident  that  to  develop  the  system  correctly 
and  with  ease,  each  team  member  needed  to  be  well  versed  in  the  selected  methodology  and  tool  set  as 
appropriate  for  the  member’s  role  on  the  team.  It  also  became  evident  that  outside  consultants  with  very 
strong  skills  in  these  areas  could  easily  be  brought  in  to  carry  out  the  exercise,  thereby  circumventing 
the  intent  of  having  specific  offeror  personnel  conduct  the  exercise.  Consequently,  in  the  suggested 
instructions  for  the  offeror,  the  team  specified  that  offeror  participation  in  the  exercise  be  limited  to 
those  key  individuals  identified  in  the  offerof  s  technical  propos^  as  part  of  the  CCPDS-R  FSD/P  team, 
that  subrantraaors  who  will  be  responsible  for  software  development  on  CCPDS-R  be  active 
participants,  and  that  consultants  be  precluded  from  participating.  To  verify  offeror  compliance  with 
these  ground  rules  and  to  assess  the  knowledge  of  individual  offeror  personnel  in  the  methodology,  the 
team  further  specified  that,  after  submission  of  the  SEE  products,  the  offerors  present  a  briefing  to  the 
Government  on  their  SEE  results,  at  which  time  the  Government  would  be  able  to  question  all  offeror 
team  members  on  their  role  in  the  exercise  and  on  specific  technical  aspects  of  the  submitted  design, 
methodology  and  tool  set  Responses  to  these  Government  questions  would  be  considered  part  of  the 
offeror's  SEE  products,  and  subject  to  evaluation  by  the  Government. 
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2.7.2.S  Deliverable  Products 


During  the  course  of  the  SEE  dry  run,  the  question  arose  as  to  what  materials  the  offerors 
should  submit  for  evaluation  and  in  what  format  the  products  should  be  delivered.  The  team  amcluded 
that,  for  those  software  architecture  components  the  offerors  chose  to  analyze  and  design,  the  offerors 
should  submit  all  requirements  analysis  and  design  pnxlucts,  both  textual  and  graphical,  that  they 
generated  as  part  of  Aeir  methodology  and  which  are  required  per  DOD-STD-2167,  as  tailored  for 
CCPDS-R.  These  products  included,  for  example,  SRSs,  STLDDs,  software  detailed  design 
documents  (SDDDs),  and  performance  analyses.  Also,  the  team  concluded  that  the  offerors  should 
submit  all  textual  products  of  the  exercise,  includitrg  requirements  atudysis  conclusions  and 
documentation,  ADL  listings,  and  other  design  ^xaimentadon  both  in  hardcopy  form  atxl  in  machine- 
readable,  9-track  tape.  The  tape  format  provided  the  Goverranent  the  capaUlity  to  browse  through  the 
text,  to  apply  certain  design  analysis  tools  to  the  ADL,  atxi  to  verify  that  the  offerors'  ADL  was 
compilable.  Finally,  the  team  concluded  that  the  offerors  should  present  a  briefing  to  the  Government 
on  their  SEE  results.  This  briefing  would  take  place  following  initial  Government  evaluation  of  the 
SEE  products  and  would  enable  the  Government  to  verify  its  rating  of  the  offerors'  SEE  performance 
and  to  assess  the  knowledge  of  the  offerors'  team  members,  as  described  in  section  2.7.2.4.  The 
instructions  for  the  offeror  were  written  to  include  these  specific  directions. 
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SECTION  3 


PLANS  FOR  EVALUATING  THE  OFFERORS 


To  evaluate  the  offerors’  perfomiance  on  the  SEE,  the  MITRE  SEE  team  developed  a  set  of 
evaluation  criteria  based  on  a  set  of  possible  discriminating  issues  found  during  the  dry  run  of  the  SEE. 
This  section  presents  a  general  overview  of  source  selection  evaluation  terminology.  It  then  describes 
the  source  selection  approach  chosen  for  the  SEE  and  delineates  how  these  discriminators  were  used  to 
derive  a  set  of  objective  source  selection  evaluation  criteria.  Next,  this  section  presents  a  description  of 
some  tools  and  techniques  selected  to  assist  the  Government  in  evaluating  the  offerors'  SEE  products. 
Finally,  this  section  details  how  the  Government  presented  the  SEE  to  the  offerors  and  how  ^ 
Government  planned  to  evaluate  the  offerors’  products. 


3.1  SOURCE  SELECTION  TERMINOLOGY  OVERVIEW 

As  defined  in  Air  Force  Regulation  (AFR)  70-15,  "Source  Selection  Policy  and  Procedures"  [5] 
and  Electronic  Systems  Division  supplement  1  to  AFR  70-15  [6],  during  source  selection,  offerors’ 
proposals  are  evtduated  against  a  set  of  predefined  criteria.  The  evaluation  criteria  are  correlated  to 
impoitant  aspects  of  the  program  which  are  significant  to  the  selection  decision  and  particulariy  to 
aspects  of  the  program  that  constitute  high  risk.  The  evaluation  criteria  are  arranged  as  evaluation  areas 
which  are  broken  down  further  into  items,  which  in  twn  may  be  broken  down  into  evaluation  factors 
and  possibly  subfactors.  The  evaluation  criteria  and  order  of  importance  are  described  to  the  prospective 
offerors  in  section  M  of  the  RFP,  however,  normally  the  evaluation  factors  and  subfactors  are  not 
identified  in  the  RFP,  section  M. 

During  source  selection,  offerors’  proposals  are  rated  against  the  evaluation  criteria  using 
predefined  standards  and  scoring  methods.  At  the  lowest  applicable  evaluation  criteria  category  (e.g., 
item,  factor,  subfactor),  standards  ate  prepared  and  used  as  positive  indicators  of  the  minimum 
performatKe  or  ccHnpliance  acceptable  to  enaUe  an  offeror  to  meet  the  requirements  of  that  evaluation 
criteria.  Thus,  standards  are  the  measures  by  which  the  Government  scores  an  offeror’s  proposal  as 
acceptable  or  unacceptable. 


3.2  SEE  SOURCE  SELECTION  APPROACH 
3.2.1  SEE  Evaluation  Criteria 

Based  on  the  dry  run,  the  team  determined  that  the  critical  issues  for  evaluating  the  CCPDS-R 
FSD/P  SEE  products  were  the  robustness  and  cohesion  of  the  offeror's  requirements  analysis, 
preliminary  design,  and  detailed  design  methodologies;  the  offeror's  familiarity  with  the  methodologies 
and  tools;  the  offeror's  Ada/software  engineering  expertise;  the  robustness,  cohesion,  and  completeness 
of  the  submitted  exercise  design;  the  offeror's  ability  to  address  and  analyze  real-time  requirements  and 
issues;  the  offeror’s  clarity  and  communication  of  design,  including  the  use  of  ADL  to  express  design; 
and  the  offeror's  compliance  with  the  SEE  system  specification  and  the  offeror's  own  SDP.  The  team 
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assessed  that  an  evaluation  of  these  issues  as  reflected  in  the  offeror's  SEE  products  would  provide 
sufficient  evidence  as  to  the  offeror’s  ability  to  design  and  develop  a  real-time  system  in  Ada  using 
modem  software  engineering  practices.  Any  other  issues  such  as  coding,  metrics,  and  full  compliance 
with  DOI>-STD-2167  were  considered  unnecessary.  Thus,  the  Government  included  only  the  above 
high-level  evaluation  criteria  for  the  SEE  in  section  M  of  the  CCPDS-R  RFP. 

Given  this  high-level  criteria,  the  Government  identified  where  the  SEE  should  be  included  in 
the  CCPDS-R  FSD/P  source  selection  process.  Since  the  CCPDS-R  source  selection  approach 
included  only  two  evaluation  areas,  technical  and  cost,  the  Government  determined  that  the  SEE  be 
included  as  one  of  the  four  items,  of  equal  importance,  in  the  technical  area.  The  Government  felt  that 
the  SEE  should  not  be  itKX>rporated  under  the  source  selection  general  considerations  area,  since  this 
area  carries  less  weight  than  the  evaluation  areas.  Also,  the  Government  concluded  that  the  SEE  item 
should  be  decomposed  into  three  factors  and  associated  subfactors  as  follows: 

a.  Factor:  methodologies 

1.  Subfactor:  requirements  analysis  methodology 

2.  Subfactor:  design  methodology 

3.  Subfactor:  interrelationship  between  requirements  analysis 
and  design  methodology 

b.  Faaor:  design 

c.  Factor:  team  expertise 

1 .  Subfactor:  methodologies 

2.  Subfactor,  team  composition 

Since  these  factors  and  subfactors  only  reflected  a  consolidation  and  reorganization  of  the  SEE  criteria 
already  contained  in  the  RFP,  section  M,  the  Government  elected  rxrt  to  include  these  factors  and 
subfactors  in  the  section  M  provided  to  the  prospective  offerors  [2]. 

3.2.2  SEE  Scoring  Method 

During  the  dry  run  of  the  SEE,  it  became  evident  to  the  team  that  an  offeror’s  failure  to  follow  a 
stated  software  engineering  approach  could  not  be  corrected  via  revision  during  the  source  selection 
period.  Thus,  for  source  selection  the  Government  decided  to  give  the  ofTerors  only  one  opportunity  to 
carry  out  the  SEE.  The  Government  instructions  stated  that  offeror  revisions  or  changes  to  SEE 
products  accomplished  after  the  conclusion  of  the  SEE  period  would  not  be  evaluated.  Further,  rather 
than  use  the  typical  color  coding  and  risk-scoring  approach  documented  in  AFR  70-15,  which  allows 
the  offerors  to  submit  proposal  revisions  in  response  to  Government  clarification  requests  and 
deficiency  reports,  the  Government  decided  that  a  unique  scoring  approach  be  used  for  the  SEE  in 
which  no  clarification  requests  or  deficiency  reports  were  employed.  The  scoring  method  defined  for 
the  SEE  was  strictly  pass/fail  with  no  risk  designated.  With  this  approach,  an  offeror  was  assessed  a 
rating  of  pass  for  the  SEE  item  if  the  offeror's  SEE  proposal  was  judged  outstanding  or  satisfactory  in 
two  of  the  three  SEE  factors;  otherwise  the  offeror  was  assessed  a  rating  of  fail.  Each  of  these  three 
SEE  factors  was  rated  as  outstanding,  satisfactory,  or  unsatisfactory.  Outstanding  indicated  that  the 
offeror  exceeded  minimum  requirements  in  a  beneficial  way  with  no  significant  weaknesses. 
Satisfactory  indicated  that  the  offeror  met  minimum  requirements  with  some  weaknesses  that  could  be 
controlled.  Unsatisfactory  denoted  that  the  offeror  fail^  to  meet  the  minimum  requirements; 
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weaknesses  could  not  be  readily  or  reasonably  corrected.  With  this  approach,  a  rating  of  fail  for  the 
SEE  did  not  tender  an  offeror  automatically  ineligiMe  for  award. 

3.2.3  Discriminator  Issues 

Prior  to  the  commencement  of  the  CCPDS-R  source  selection  technical  evaluation,  die  MITRE 
SEE  team  developed  preliminary  standards  for  eadi  of  the  above  SEE  factors  and  subfactors  based  in 
part  on  the  recommended  evaluation  criteria  and  a  set  of  lower  level  discriminating  issues  identified 
during  the  course  of  the  SEE  dry  nm.  The  lower  level  discriminators  related  to  identificatirai  of 
specification  ambiguities,  allocation  of  timing  requirements  across  system  components,  behavioral 
aspects  of  the  exercise  system,  interface  specification,  and  consistent  representation  of  design 
information  across  ADL,  text  and  graphics. 

3.2.3.1  Specification  Ambiguities 

In  the  dry  run  of  the  exercise,  the  team  discovered  instances  of  incompleteness  and  ambiguities 
in  the  draft  exercise  specification.  Many  of  these  instances  were  uncovered  during  requirements 
analysis  cmly  after  discussion  among  the  team  members;  initially,  each  member  thought  he  or  she 
understood  the  intent  of  the  requirements  and  only  when  two  members  had  to  agree  did  the 
incompleteness  become  apparent  Many  of  these  ambiguities  were  impossiUe  to  resolve  fully  until 
derived  requirements  were  presented  at  the  SRS  level.  An  examf^C  is  the  interpretation  of  the 
requirement  that  the  exercise  system  wUl  "simulate  the  CCPDS-R  missile  warning  capability  in  real- 
time"  (see  t^rpendix  A).  Other  areas  of  incom[4eteness  related  to  the  difficulty  of  stating  performance 
requirements  concisely;  for  example,  the  requirement  that  the  "time  from  completion  of  [data]  entry  [by 
the  user]  until  the  database  is  modified  to  reflect  the  update  shall  not  exceed  two  seconds"  (see 
appendix  A).  Such  requirements  force  end-to-end  performance  measurement  across  different 
components.  To  prevent  inconea  interpretaticms  of  specificaticm  ambiguities  from  having  later 
catastrophic  and  costly  results,  it  is  imperative  that  the  methodology  emfdoyed  for  requirements  analysis 
itKlude  approaches  for  detecting  and  resolving  specification  amlxguities  arxi  itKonsisterxnes.  The 
offerors'  SEE  products  were  therefore  expected  to  reflect  a  thorough  identification  and  resolution  of 
specification  ambiguities. 

3.2.3.2  Timing  Requirements 

As  a  result  of  the  dry  run,  the  team  found  that  allocation  of  timing  budgets  to  software 
components  for  the  SEE  was  very  difficult  to  support  analytically.  As  mentioned  above,  this  was  due 
in  part  to  inherent  problems  in  stating  quantitative  perfoimance  requirements  in  a  rigorous,  testable 
manner.  But  the  primary  problem  was  due  to  the  nature  of  the  exercise:  the  analysis  to  support  timing 
budget  allocation  requires  simulation  and/or  (xototyping  activities,  and  the  tools  smd  time  ne^ed  to  do 
this  were  not  available  to  the  team  during  the  exercise  period.  The  team  also  found  that  timing  analyses 
must  be  done  during  the  requirements  analysis  phase  to  do  a  proper  allocation  of  requirements.  The 
offerors'  SEE  results  were  therefore  expect^  to  irK^lude  an  ex{dicit  performance  an^ysis  activity,  done 
during  requirements  analysis,  which  would  provide  input  to  the  SRSs. 

3.2 Behavioral  Aspects  of  the  System 

A  number  of  technical  questions  arose  during  the  exercise  dry  run  that  related  to  the  correct, 
reliable  operation  of  the  system.  The  team  felt  that  these  issues  should  be  addressed  in  the  preliminary 
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design  by  means  of  exfdicit  use  of  Ada  language  features.  These  issues  were  the  dear  identification 
(from  the  AIX  and  the  graphical  representation)  of  the  major  control  tiueads  nmning  throughout  the 
system:  the  synchronization  and  prioritization  of  concurrent  tasks;  tiie  avoidance  of  system-wide 
(kadlock;  the  effectiveness  of  the  mechanisms  used  to  initialize  and  terminate  the  exercise  system  (these 
mechanisms  can  be  implidt  via  reliance  on  Ada  dabonuion  order  or  can  involve  explicitly  imfdemented 
procedures);  and  the  effective  use  of  Ada  exception  haixlling. 

3. 2.3. 4  Interfaces 

Interface  consistency  has  typically  leagued  DOD  software  development  effoits  over  the  years, 
and  the  advantages  of  Ada  for  producing  consistent  interface  package  spedfications  are  obvious.  While 
the  team  did  not  really  expect  that  an  offeror  would  fail  to  use  Ada  properly  for  data  definition  on  such  a 
small  exercise,  the  team  felt  nevertheless  that  effective  use  of  Ada  ^uld  be  demonstrated  in  the  SEE 
products. 

3.2.3.5  Consistent  Representation 

The  SEE  system  specification  requires  that  both  graphical  representation  arxl  ADL  be  used  to 
describe  design  information  and  that  they  be  employed  consistently.  The  team  found  in  the  SEE  dry  run 
that  graphical  representations  are  necessary  and  useful  to  depict  tc^level  and  detailed  views  of  the 
software  architecture  as  well  as  relationships  amtmg  components.  ADL  is  then  used  to  fill  in  details  and 
to  enhance  definitions.  The  team  found  that  these  techniques  must  supplement  one  artother  since  they 
lose  effectiveness  if  used  to  describe  different  things;  consequently,  the  offerors’  SEE  products  were 
expeaed  to  reflea  compatible  ADL  and  graphical  design  representations. 


3.3  EVALUATION  TOOLS  AND  TECHNIQUES 

Given  the  rather  low  level  of  detail  described  above  against  which  the  offerors'  SEE  products 
would  be  evaluated  together  with  the  potentially  large  amount  of  data  to  be  submitted  by  the  offerors, 
the  SEE  team  identified  the  possible  need  for  some  additional  tools  and  techniques  to  assist  the 
Government  in  evaluating  die  SEE  products.  To  that  end,  the  team  recommend  that  the  Government 
use  the  ESD  acquisition  support  environment  (EASE)  and  a  set  of  evaluation  checklist  questions  for  the 
SEE  source  selection. 

3.3.1  EASE 

EASE  is  a  prototype  workstation-based  tool  intended  to  support  Government  review  of  contract 
technical  documentation.  Specifically,  EASE,  which  was  under  development  at  the  time  of  the  SEE  dry 
mn,  is  oriented  towards  the  review  of  contraaor  products  relating  to  the  acquisition  of  Ada  software. 
These  products  will  primarily  consist  of  ADL  and  Ada  code.  At  maturity,  EASE  will  support  a  wide 
range  of  analytic  activities,  itKluding  RFP  preparation,  modeling,  requirements  analysis,  design 
analysis,  and  tool  assessment  EASE  is  not  intended,  however,  to  support  management  ftinaions. 

The  EASE  prototype  executes  on  a  Sun-3  UNIX®  ^ -based  workstation.  EASE  takes  full 


1 .  UNIX®  is  a  trademark  of  AT&T  Bell  Laboratories. 
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advantage  of  the  Sun's  large  bit  mapped  display  and  windowing  system.  Different  tools  execute  in  their 
own  windows,  and  infoimation  is  managed  in  a  common  database  hidden  from  the  user.  At  the  time  of 
the  CCPDS-R  source  selection,  die  only  tools  integrated  with  EASE  were  the  GNU  Emacs  editor,  the 
Verdix  Ada  compiler  and  several  utilities  delivered  with  the  compiler. 

For  the  SEE,  the  team  proposed  the  use  of  EASE  specifically  for  browsing  duough  the 
offerors'  textual  products  and  for  assisting  in  the  evaluation  of  the  ADL  submitted  with  the  products. 
Since  the  SEE  system  specification  required  that  the  offerors'  design  be  documented  in  compilable 
ADL,  the  team  recommended  that  the  Government  use  EASE  to  test  whether  the  offerors'  ADL  did  in 
fact  compile.  The  Government  elected  to  follow  these  recommendations. 

3.3.2  Checklist  Questions 

In  addition  to  the  use  of  EASE,  the  SEE  team  recommended  that  a  set  of  informal  checklist 
questions  be  employed  to  assist  the  SEE  evaluators  in  their  rating  of  the  offerors'  SEE  products  against 
the  factors  and  standards.  The  questions  would  be  correlated  with  specific  factors  and  standards  and 
would  highlight  particular  issues  which  must  be  addressed  to  determine  if  a  standard  is  met.  The 
questions  would  serve  two  purposes;  for  those  source  selection  evaluators  who  participated  in  the  dry 
run,  the  questions  would  serve  as  reminders  of  key  points  to  look  for  in  the  SEE  products,  and  for 
those  evrduators  who  were  not  familiar  with  the  SEE  prior  to  source  selection,  the  questions  would 
serve  as  a  checklist  for  evaluating  the  products  and  determining  whether  or  not  standards  had  been  met 
In  support  of  this  recommendation,  the  SEE  team  prepared  an  extensive  list  of  evaluation  questions  to 
serve  as  a  basis  for  the  checklist. 


3.4  RELEASE  OF  SEE  TO  OFFERORS 

Based  upon  MITRE's  dry  tun  of  the  SEE,  it  was  expected  that  both  the  offeror  preparatim  of 
the  SEE  and  the  Government  evsduation  of  the  resulting  products  would  be  intensive  and  time 
consuming.  Fuithermore,  the  Government  resmirces  to  review  the  SEE  products  would  be  limited, 
since  the  evaluators,  about  eight  people,  would  most  likely  be  re^nsible  for  reviewing  both  the  SEE 
products  and  the  offerors'  technical  proposals.  Therefore,  to  give  the  Government  time  to  review  the 
offerors'  CCPDS-R  technical  proposals  and  SDI^  as  well  as  to  give  the  offerors  adequate  time  to 
prepare  both  the  technical  proposals  and  the  SEE,  the  Government  elected  i»t  to  begin  the  SEE  until 
after  receipt  of  the  offerors'  technical  proposals  and  SDPs.  Consequently,  the  Government  did  not 
include  the  SEE  system  qiecification  and  detailed  instructions  for  the  offeror  in  the  RFP  released  on 
10  October  1986;  it  only  included  a  copy  of  the  S^  section  M  evaluation  criteria  and  a  preliminary  set 
of  SEE  instructions  for  the  offeror  in  the  RFP  instructions  for  proposal  preparation  (IFI¥).  As  stated 
in  section  3.2.1,  the  SEE  secticm  M  evaluation  criteria  identifi^  the  basis  on  which  the  offerors'  SEE 
products  would  be  judged.  The  preliminary  instructions  for  the  offeror  contained  the  general  ground 
mles  for  the  conduct  of  the  SEE  and  a  brief  description  of  the  SEE  products  to  be  generated  and 
submitted  by  the  offerors  for  Government  evaluatioa  Upon  recei{M  of  the  offerors'  technical 
proposals,  due  on  10  November  1986,  the  Government  planned  to  supply  each  offeror  the  actual  SEE 
system  specification  and  the  detailed  instructions  for  the  offeror.  Figure  S  shows  the  interaction  of 
Government  and  offeror  activities  during  the  timeframe  of  source  selection.  Copies  of  die  SEE  system 
^lecification  and  detailed  instructions  for  the  offeror,  the  RFP  IFIT  preliminary  instructions  for  the 
offeror,  and  the  RFP  section  M  may  be  found  in  appendices  A,  B,  a^  C,  respectively. 
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!.  Government  and  Offeror  Activities  During  the  Source  Selection  Period 


3.5  GOVERNMENT  EVALUATION  APPROACH 

The  Government's  planned  approach  for  evaluating  the  offerors'  SEE  products,  delivered  3  1/2 
weeks  after  receipt  of  the  SEE  system  specification  and  detailed  instnictions  for  foe  offeror,  consisted  of 
a  first-pass  evaluation,  an  in-house  audit  at  each  offeror's  facility,  and  a  completed  evaluadoa  The 
Government's  intent  was  that  upon  receipt  of  the  offerors'  SEE  products,  foe  Govenunent  would 
perform  a  preliminary  evaluation  of  the  products,  allocating  approximately  one  week  for  each  offeror. 
Following  that  evaluation  period,  the  Government  would  conduct  a  1  -day  audit  at  each  offeror's 
facility.  The  offeror's  SEE  products  and  results  of  foe  audit  would  then  be  factored  into  a  final 
evaluation  to  be  completed  by  the  Government  within  a  week  of  foe  audit  The  following  paragraphs 
describe  the  process  of  the  first-pass  evaluation  and  the  audit 

3.5.1  First-Pass  Evaluation 

The  purpose  of  the  first-pass  evaluation  was  to  obtain  a  preliminary  assessment  of  each 
offeror's  performance  on  the  SEE  and  to  identify  strengths  and  weaknesses  in  the  offeror's  SEE 
products.  The  Government  would  carry  out  the  first-pass  evaluation  by  scoring  each  offeror's  SEE 
products  against  the  predefined  source  seleaion  factors  and  standards.  For  each  offeror,  the 
Government  would  prepare  draft  documentation  which  would  describe  the  offeror's  SEE  products,  foe 
offeror's  strengths  and  weaknesses  relative  to  the  factors  and  standards,  and  an  overall  assessment  of 
foe  offeror's  performance  on  foe  SEE.  Also,  the  Government  would  prepare  a  set  of  questions  tailored 
for  each  offeror  which  would  be  posed  to  foe  offeror  during  the  on-site  audit.  The  intent  of  these 
questions  was  to  verify  foe  Government's  interpretation  and  evaluation  of  the  SEE  products  and  to 
assess  foe  offeror's  SEE  team  ctqrabilities  in  software  engineering.  Ada,  and  the  selected  methodologies 
and  tools. 

3.5.2  Audit 

As  stated  above,  the  purpose  of  foe  Government  audit  at  each  offeror's  facility  was  to  verify  the 
Government's  preliminary  assessment  of  foe  offeror's  SEE  products  and  to  obtain  additional 
information  to  complete  its  evaluation.  As  plaruied,  foe  in-house  SEE  audit  would  consist  of  two  parts: 
an  offeror  briefing  and  a  question  and  answer  sessioa  The  brieflng  would  provide  an  c^rportunity  for 
foe  offeror  to  exi^ain  the  methodology  proposed  for  CCPDS-R  and  emf^oy^  on  the  Sffi.  The  briefing 
would  include,  at  a  minimum,  a  summary  of  the  offeror's  management  approach,  an  overview  of  the 
requirements  analysis  approach,  an  overview  of  foe  preliminary  and  detailed  design  ^roaches,  an 
identification  of  any  assumptions  made  while  carrying  out  foe  SEE  and  generating  the  SEE  products, 
and  an  identification  of  any  deviations  made  from  the  SDP  along  with  the  rationale  for  those  deviatirais. 

The  briefing  would  not  include  any  discussion  of  further  work  which  the  offeror  may  have 
completed  following  foe  submission  of  the  SEE  products,  since  foe  Government  would  rxrt  evaluate  this 
additional  work.  The  question  and  answer  session  would  provide  an  opportunity  for  the  Government 
to  obtain  clarifying  information  about  foe  offeror's  SEE  prmlucts  and  to  query  individual  offeror  team 
members  spontaneously  to  test  their  expertise  with  the  selected  methodologies  and  tools.  Each  member 
of  the  offeror's  SEE  team  would  be  required  to  be  present  during  the  audit  to  respond  to  specific 
questions  directed  to  that  individual,  lire  Government  would  maintain  a  transcript  of  foe  questions  atxl 
answers.  This  transcript  together  with  the  briefing  presentation  material  and  the  SEE  products  delivered 
at  foe  end  of  the  3  1/2-week  exercise  period  would  be  considered  part  of  foe  offeror's  proposal  atvl 
included  in  the  Government's  final  evaluation  of  foe  offeror's  SEE  results. 
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SECTION  4 


DRY  RUN  LESSONS  LEARNED 


As  a  result  of  the  MITRE  dry  run  of  the  SEE,  numerous  lessons  were  learned.  These  lessons 
may  be  divided  into  two  categories:  administrative  issues  that  relate  to  defining,  oi^ganizing  and 
including  the  SEE  as  part  of  the  source  selection  process,  and  technical  issues  that  concern  software 
development  and  Ada  in  general.  Lessons  learned  relating  to  administrative  issues  have  been  described 
throughout  sections  2  and  3.  This  section  summarizes  the  major  technical  lessons  learned  relating  to 
software  engineering  and  specifically  requirements  analysis,  object  oriented  design,  Ada,  and  DOD- 
STD-2167.  It  also  highlights  how  those  lessons  were  factored  into  either  the  CCPDS-R  FSD/P 
program  and/or  the  standards  prepared  for  the  CCPDS-R  source  selection. 


4.1  REQUIREMENTS  ANALYSIS 

During  the  course  of  MITRE's  dry  run  of  the  SEE,  the  team  made  observations  regarding  the 
time  allocated  for  requirements  analysis.  Government  interaction,  a  formal  requirements  analysis 
methodology,  and  the  completion  of  the  requirements  analysis  phase. 

4.1.1  Time  Allocation 

The  team  discovered  that  much  more  time  than  anticipated  was  needed  to  produce  a  thorough 
requirements  analysis  and  associated  documentatioa  As  reflected  in  secticm  2.3,  ^  team  spent 
approximately  thr^  times  longer  on  the  requirements  analysis  effort  than  originally  planned.  This  extra 
time  was  due  to  the  following  conditions: 

a.  The  team  members  initially  assumed  that  the  requirements  in  the  exercise  specification  were 
clear  and  would  not  require  extensive  analysis; 

b.  The  original  1-week  allottment  for  requirements  analysis  was  overly  optimistic  but  was 
necessary  to  achieve  the  scheduled  IS  July  1986  RFP  release; 

c.  The  team  lacked  a  foimal  approach  to  requirements  analysis  at  the  start  of  the  exercise;  and 

d .  The  DOD-STD-2 167  SRS  DID  was  new,  requiring  a  learning  curve,  and  it  specified  a 
lower  level  of  detail  than  anticipated  by  the  team  members. 

Eliminating  the  extra  time  spent  due  to  coixlitions  a  through  c  above,  it  was  estimated  that  as 
much  as  twice  the  originally  scheduled  time  was  spent  on  requirements  analysis  due  to  the  DOD-STD- 
2167  required  level  of  detail.  Based  on  this  observation,  the  Government  developed  a  projected 
CCPDS-R  FSD/P  phase  schedule  which  inducted  approximately  one  additional  month  for  requirements 
analysis  beyotKl  that  typically  estimated.  Also,  the  Government  elected  to  scrutinize  carefully  during 
source  selection  the  offerors'  proposed  CCPDS-R  software  development  schedules  to  ensure  that 
adequate  time  had  been  allocated  for  requirements  analysis. 
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4.1.2  Government  Interaction 

During  the  dry  nin  of  the  SEE,  the  team  observed  that  the  presence  of  a  "Government" 
lepiesentative  during  the  requirements  analysis  effort  greatly  facilitated  progress  during  that  phase. 

This  person  was  able  to  assist  the  development  team  by  clarifying  ambiguities  and  identifying  iraxirrect 
assumptions.  As  mentioned  in  section  2.1.22,  the  team  recognized  that  the  Government  could  not  play 
a  similar  role  during  the  offerors'  execution  of  the  SEE.  However,  based  upon  this  SEE  observation, 
the  Government  elected  to  include  in  the  CXTPDS-R  statement  of  work  (SO^  a  provision  for  the 
Government  to  maintain  a  representative  on-site  in  the  contractor's  facUity  throughout  the  requirements 
analysis  phase  to  monitor  the  contractor's  effort  and  to  assist  in  obtaining  responses  to  contractor 
questions. 

4.1.3  Formal  Methodology 

The  SEE  team  found  that  a  formal  methodology  was  needed  during  the  requirements  analysis 
phase  to  ensure  a  thorough  analysis  of  the  specification  functional  arrd  performance  requirements,  arxl 
to  maintain  control  over  the  effort.  By  formal  methodology,  it  is  meant  that  the  methodology  be 
documented,  that  it  cover  the  areas  requited  in  DOIXSTD-2167,  and  that  it  be  teachable.  Failure  to  have 
such  a  formal  methodology  in  f^ace  a^  one  with  which  the  team  was  well  versed,  caused  initial  delays 
in  the  MITRE  dry  run.  Based  on  this  observation,  the  Government  decided  to  examine  the  offeror's 
SEE  products  to  assess  the  presence  of  a  formal  requirements  analysis  methodology  in  which  the 
offeror’s  SEE  team  was  knowledgeable. 

4.1.4  Completion  of  Phase 

The  SEE  dry-run  team  noted  that  it  was  difficult  to  determine  where  requirements  analysis 
stopped  and  design  started,  especially  given  the  requirements  of  DOD-STD-2167.  For  example,  the 
DOD-STD-2167  requirements  for  the  software  requirements  qxcification  imply  that  design  information 
related  to  timing  and  sizing  be  included  in  the  SRS  (see  section  4.4),  which  is  produced  during  the 
requirements  analysis  phase.  It  was  determined  that  by  having  a  formal  requirements  analysis 
methodology  and  a  formal  design  methodology,  and  by  tailoring  DOD-STD-2167,  this  pr^lem  would 
be  minimiz^.  For  the  source  selection  therefore  the  Government  planned,  first,  to  scrutinize  the 
offeror's  requirements  analysis  and  design  methodologies  to  ensure  that  they  were  robust  formal 
methodologies,  and,  second,  to  examine  the  interrelationship  of  the  two  methodologies  to  ensure  that 
they  were  compatible,  with  clearly  defined  steps  for  transitioning  from  one  phase  to  the  next 


4.2  OBJECT-ORIENTED  DESIGN 

As  mentioned  in  section  2.2,  the  MITRE  SEE  team  selected  OOD  as  defined  by  Grady  Booch 
for  the  methodology  to  be  used  during  the  design  phase.  Booch's  OOD  attempts  to  map  solutions 
directly  to  the  problem  as  viewed  in  real-world  terms,  forcing  a  proMem  set  to  be  view^  in  terms  of  a 
set  of  software  objects,  each  with  its  own  set  of  applicable  operations.  Booch's  OOD  consists  of  three 
phases:  problem  definition,  informal  strategy,  and  formal  strategy.  The  first  phase  consists  of  defining 
the  problem  in  English  at  a  high  level,  the  goal  being  to  gain  an  u^erstanding  of  the  structure  of  the 
problem  space.  The  second  phase  consists  of  developing  an  informal  strategy  wherein  natural  English 
descriptions  of  the  problem  space  are  used  to  narrate  the  problem.  The  goal  of  the  second  phase  is  to 
continue  expanding  the  designers'  understanding  of  the  problem  without  limiting  their  ability  to  think 
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about  the  problem  and  without  concerning  themselves  with  die  stnicture  of  the  solution.  Finally,  the 
third  phase  consists  of  formalizing  the  strategy.  Uang  the  informal  strategy  developed  in  the  second 
phase,  nouns  and  verbs  are  extracted  and  become  the  objects  and  operations  in  the  solution.  The  nouns 
are  used  to  imply  abstract  data  types  and  specific  real-world  objects.  The  verbs  are  used  to  define  real- 
world  operations  with  particular  objects.  Also,  adverb  phrases  are  extracted  to  identify  attributes  of  the 
operations,  and  interfaces  between  objects  are  described.  Finally,  the  operations  previously  identified 
for  each  object  are  implemented  in  executable  form  (e.g.,  AIX.).  This  process  is  repeated  until  a  point 
is  reached  where  the  level  of  decomposition  is  understandable  without  furtiier  modularity. 

During  die  dry  run  of  the  SEE,  it  appeared  to  die  team  that  Booch's  OOD  was  an  incomi^ete 
methodology.  While  it  provided  practical  guidance  for  object  identification,  it  lacked  suf^rt  for 
requirements  traceability  and  completeness,  performance  analysis,  concurrency  (i.e.,  multitasking), 
initialization  and  termination  additions,  and  error  detection  arid  handling.  It  did  not  clearly  specify 
how  to  transition  from  requirements  analysis  to  design  nor  did  it  specify  guidelines  for  the  completion 
of  detailed  design.  Furthermore,  the  team  discovered  that  the  use  of  OOD's  informal  strategy  was  non¬ 
productive  in  practice.  To  have  the  SRS  and  then  have  to  write  the  informal  strategy  resulted  in 
duplications  of  effort  The  team  observed  that  in  many  cases  the  informal  strategy  could  be  develqied 
so  as  to  produce  contrived  results. 

As  an  outcome  of  these  observations,  the  Government  included  specific  SEE  factors  and 
standards  to  ensure  that  the  offerors'  design  metlKxiologies  were  comi^ete,  robust  and  that  they 
contained  specific  procedures  to  resolve  the  above  issues.  In  particular,  the  Government  plann^  to 
assess  whether  the  offerors'  design  methodologies  contained  clearly  defined  procedures  for 
transitioning  between  requirements  analysis  and  design  phases  and  for  harxlling  initialization  and 
termirution,  exception  handling,  concurrency,  and  performance  analysis. 


4.3  ADA 

While  dry  rtinning  the  SEE,  the  team  ideruified  several  observations  relatmg  to  both  the 
technical  and  management  aspects  of  Ada.  These  issues  concerned  control  flow,  ADL,  and  personnel. 

4.3.1  Flow  of  Control 

In  a  multitasking  system,  many  control  flow  issues  must  be  resolved.  These  issues  typically 
relate  to  shared  access  to  resources,  and  include  deadlock,  process  starvation,  race  conditions, 
serialization,  and  synchronizatioa  The  team  spent  considerable  time  addressing  deadlock  and  process 
starvation,  in  parti^ar. 

Deadlock  conditions  exist  when  two  or  more  processes  cannot  execute  because  eadi  is  waiting 
for  a  resource  held  by  another,  similarly,  process  starvation  occurs  when  a  process  is  waiting  to  access 
a  resource  and  the  scheduling  mechanisms  either  never  service  the  process'  request,  or  result  in  an 
unpredictably  long  delay.  In  the  usual  sequential  programming  languages,  operating  systems  and  cyclic 
executives  consider  and  handle  most  control  flow  issues;  thus,  only  the  limit^  group  of  operating 
system  programmers  need  address  these  issues.  However,  while  dry  running  the  SEE,  the  team 
discovned  that,  even  at  the  apfdication  level,  the  Ada  tasking  construct  must  be  used  with  great  care  to 
avoid  deadlock  and  process  starvation;  consequently,  all  applications  software  designers  and 
programmers  must  address  these  issues.  To  address  such  conditions  within  the  applications  software, 
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the  software  development  methodology  must  include  techniques  for  designing  effective  controls  for  the 
detection  and/or  prevention  of  deadlock  and  process  starvation. 

During  the  dry  run,  these  issues  were  dealt  with  in  part  through  the  use  of  cantxiical  task  idioms 
and  strategies  that  provided  controlled  access  to  shared  resources.  Given  the  team's  conclusion  about 
the  importance  of  these  control  issues  and  the  fact  foat  the  SEE  subsystem  was  to  be  designed  using 
ADL,  the  Government  chose  to  consider  during  source  selection,  as  part  of  the  completeness  and 
robusmess  of  the  offerors'  design  methodologies,  the  ability  of  the  offerors'  design  methodologies  to 
address  flow  control,  in  general,  and  deadlock  and  process  starvation,  in  particular. 

4.3.2  ADL 

The  MITRE  SEE  team  designed  the  SEE  system  using  ADL,  documented  the  design  with  the 
ADL  incorporated  into  the  DOD-STD-2167  products  prepared  by  the  team,  and  presented  the  design  to 
the  "Government"  representatives  via  the  ADL  at  a  mock  PDR.  TTie  basic  conclusion  the  team  reached 
from  these  efforts  was  that  the  use  of  ADL  by  itself  does  not  present  a  global  picture  of  the  entire 
system  to  the  developers.  In  its  design  meetings,  the  team  came  to  rely  on  Buhr  diagrams  as  the 
primary  design  representatiort  Moreover,  the  team  found  that  presenting  only  ADL  at  the  dry-nm  PDR 
failed  to  convey  design  information  clearly  to  all  reviewers.  As  a  result  of  this  coiKlusion,  the 
Government  modified  the  CCPDS-R  system  specification  to  require  the  use  of  gr^cal  noution  to 
convey  design  information  in  conjunction  with  the  ADL.  Furthermore,  since  the  SEE  system 
specification  contained  the  same  graphical  notation  requirements  as  the  CCT*DS-R  system  specification, 
the  Government  opted  to  consider  during  source  selection,  as  part  of  the  completeness  and  robustness 
of  the  offerors'  design  methodologies  and  clarity  and  communicatiwi  of  design,  the  offerors'  gr^hical 
notation  to  ensure  that  it  was  well-defined,  it  was  coisistent  with  the  ADL,  and  it  contained  enough 
features  to  convey  the  information  available  via  tire  ADL  constructs. 

4.3.3  Personnel 

During  the  course  of  dry  running  the  Sffi,  the  team  encountered  several  issues  related  to  Ada 
and  personnel.  These  consisted  of  personnel  training,  retentiw)  of  Ada-trained  staff,  aixl  preseiree  of 
Ada  experts  on  development  teams. 

4.3.3.1  Training 

The  team  observed  during  the  dry  nm  that  Ada  training  must  occur  at  all  levels  of  the  software 
development  and  acquisition  teams;  from  users,  programmers,  and  designers,  to  program  managers  and 
reviewers.  The  team  also  noted  that  training  for  Ada  programmers  and  designers  is  slower  and  more 
difficult  than  training  for  other  programming  languages,  primarily  because  Ada  imposes  tire  software 
engineering  discipline  of  a  methodology  on  its  users.  To  a  greater  extent  than  in  other  langurs,  an 
Ada  programmer  must  be  a  software  engineer  arxl  must  be  knowledgeable  of  the  methodologies 
employed,  the  graphical  notation  used,  ADL,  and  the  Ada  language  itself.  The  Ada  [reograiiuner  must 
be  well  versed  in  all  these  issues  at  all  stages  of  development;  simf^y  learning  Ada  syntax  and  semantics 
is  not  enough.  Based  on  the  SEE  dry-run  results,  the  team  estim^ed  that  training  for  Ada  could  be  at 
least  two  to  three  times  longer  than  for  other  languages.  Fmally,  the  team  concluded  that  the  SEE  dry 
run  served  as  an  excellent  vehicle  to  teach  Ada  as  well  as  software  engiireering  and  software  acquisitioa 
The  SEE  dry  run  served  as  a  far  more  substantive  approach  for  teaching  software  engitreering  and  Ada 
than  the  typical  5-day  courses  offered  in  these  areas,  which  usually  concentrate  only  on  theory  rather 
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than  practical  ai^cations.  Specifically,  the  SEE  dry  lun  provided  team  members  hands-on  training  in 
all  aspects  of  software  development,  including  Ada,  meth^logies,  DOD-STD-2167,  requirements 
analysis,  design,  and  software  management  The  one  major  limitation  of  die  SEE  dry  nin  was  that  it 
did  not  cover  the  complete  software  development  cycle  since  it  did  not  progress  all  the  way  through 
code  and  testing. 

As  a  result  of  the  above  lessons  learned,  the  Government  decided  to  examine  the  CCPDS-R 
FSD/P  offerors  to  ensure  that  the  offerors'  companies  provided  in-depth  Ada  training  which  was  geared 
for  all  offeror  personnel  associated  with  CCPDS-R  software  development,  as  appropriate  for  assigned 
roles,  and  which  exceeded  the  typical  1-  to  5-day  courses.  Furthermore,  the  Government  made  plans  to 
train  its  own  CCPDS-R  project  personnel  in  Ada  after  source  selection  by  rerunning  the  SEE  ftcm 
requirements  analysis  through  testing,  with  select  project  personnel  serving  as  the  team  members,  and 
by  having  all  projea  individuals  participate  in  at  least  some  typical,  formal  Ada  courses,  as  ^propriate 
for  their  given  roles  and  responsibiUties. 

4J.3.2  Retention  of  Ada-Trained  Staff 

One  of  the  primary  risks  associated  with  Ada  today  is  the  ability  to  obtain  and  retain  highly 
qualified  Ada  engineers,  since  the  number  of  such  individuals  is  extremely  small.  The  MITRE  SEE 
'  earn  itself  experienced  problems  in  these  areas  during  the  dry  run  of  the  SEE.  MITRE  had  difficulty  in 
assembling  a  sufficient  number  of  Ada-trained  peo(4e  who  could  devote  a  signiftcant  amount  of  time  to 
the  SEE,  and  the  SEE  team  itself  experienced  the  (teparture  of  one  of  the  technical  leads.  As  a  result  of 
confinning  this  Ada  risk  during  the  SEE  dry  run,  the  Government  decided  to  assess  each  CCPDS-R 
offerot^s  ability  to  assemble  and  manage  a  team  of  engineers  for  the  SEE  who  were  well  versed  in  Ada, 
as  well  as  the  proposed  methodologies,  software  development  tools  and  procedures,  and  ADL.  In 
addition,  to  help  deter  the  departure  of  the  FSD/P  OHitractor's  key  Ada/software  engineering  personnel, 
the  Government  included  an  award  fee  plan  in  the  CCPDS-R  FSD/P  model  contract  [2].  The  contract 
requires  that  the  contractor  flow  down  ^  percent  of  the  award  fee  directly  to  the  contractor  employees 
working  on  CCPDS-R.  and  rxit  to  the  company  as  a  whole.  As  defined,  the  award  fee  is  tied  to  the 
successful  completion  of  specific  CCPDS-R  milestones. 

4.3.3.3  Ada  Expert 

As  mentioned  in  section  2. 1 ,  the  MITRE  SEE  team  included  two  Ada/software  engineering 
experts  who  were  familiar  with  all  aspects  of  Ada  and  particularly  its  more  complex  constmcts.  These 
individuals  had  significant  software  development  experience  as  well  as  a  deep  understanding  of  the 
relevant  and  often  more  complex  software  engineering  issues.  The  preserve  of  these  two  experts  was 
cmcial  to  the  SEE  development  progress.  They  served  as  mentors  to  the  rest  of  the  team  and  as  such 
were  able  to  keep  the  rest  of  the  team  on  track,  to  point  out  areas  overlooked  by  the  team  members,  and 
to  answer  or  resolve  detailed  software  engineering  arxi  Ada  questions. 

Given  the  importarve  of  these  Ada  experts  on  the  MITRE  SFE  team,  the  Government 
concluded  that  the  software  development  risks  on  a  comf^ex  Ada  development  could  be  substantially 
reduced  if  the  contractor’s  team  included  at  least  one  or  more  strong  Ada  technical  leads/experts  who 
were  well  versed  in  all  the  detailed  aspects  of  Ada  and  software  engineering.  As  a  result  of  this 
observation,  the  Government  elected  to  consider,  as  part  of  the  offeror’s  team  expertise,  the  offeror’s 
ability  to  organize  a  SEE  team  that  included  strong  Ada/software  engineering  technical  leads. 
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4.4  DOD-STD-2167 


DOD-STD-2167  defines  a  software  development  process  which  is  apf^cable  throughout  the 
system  life  cycle.  The  system  life  cycle  is  divided  into  phases.  Each  f^iase  has  associated  with  it  one  or 
more  products  to  be  generated,  and  it  culminates  in  a  review  or  audit  The  products  required  at  each 
I^ase  may  consist  of  preliminary  or  completed  products.  DOD-STD-2167  is  inteixled  to  be  tailored  for 
each  particular  application,  as  necessary.  Figure  6  illustrates  the  DOD-STD-2167  products  and  reviews 
with  each  of  the  phases  as  tailored  for  Ae  CCPDS-R  effort,  along  with  an  identification  of  those 
DOD-STD-2167  products  and  reviews  considered  applicable  to  the  SEE. 

The  SEE  team  performed  the  dry  run  in  accordance  with  DOD-STD-2167,  as  tailored  for 
CCPDS-R.  As  the  dry  run  progressed,  the  team  observed  that  the  already  existing  CCPDS-R  tailoring 
of  DOD-STE>-2167  requited  further  tailoring  because  of  what  the  team  considered  inappropriate 
requirements  of  the  standard.  For  example,  the  DOD-STD-2167  DID  for  the  SRS  requires  data  which 
seems  premature  and  in  some  cases  impossible  to  obtain  during  the  requirements  analysis  stage  of 
development.  In  particular,  the  input,  processing,  and  output  sections  of  the  SRS  DID  require  the 
specification  of  items  such  as  units  of  measure  and  ranges  for  inputs  and  outputs,  the  exact  intent  of  the 
operation,  error  detection,  and  algorithms.  However,  the  team  fourxl  that  during  the  requirements 
analysis  phase,  units  of  measure  at  this  level  may  be  impossible  to  define  and  that  delineation  of  the 
processing  section  seemed  to  force  the  conceptualization  of  a  design,  which  contradicts  the  intent  of  the 
requirements  arudysis  effort  The  SRS  DID  ^so  requires  the  specification  of  timing  and  sizing  data 
against  which  the  software  will  be  tested,  since  the  SRS  is  the  baseline  document  for  software  formal 
qualification  testing  to  the  Government  For  current  Ada  developments,  this  is  almost  impossible  to  do, 
since  previous  data  on  programs  developed  in  Ada  is  minimal.  Thus,  the  team  determined  that  any 
timing  arxl  sizing  estimates  entered  into  an  SRS  during  the  requirements  analysis  phase  for  an  Ada 
development  were  especially  weak;  the  possibility  was  extremely  hi^  that  the  timing  and  sizing  data 
contained  in  an  authenticate  SRS  would  hold  rx)  validity  later  in  the  development  effort 

As  a  consequence  of  these  observations,  the  Government  made  further  modifications  of  its 
tailoring  of  DOD-STD-2167  as  contained  in  the  CCPDS-R  SOW.  The  tailoring  included,  for  example, 
the  approach  of  only  baselining  the  sizing  and  timing  estimates  contained  in  an  SRS  at  the  software 
specification  review,  but  not  finalizing  these  estimates  until  system  PDR,  at  which  time  the  contractors, 
through  their  performance  analysis,  design,  and  prototyping  efforts  would  have  substantial  evidence  to 
support  these  estimates.  A  complete  tailoring  of  DOD-STD-2167  for  CCPDS-R  may  be  found  in  the 
CCPDS-R  FSD/P  Statement  of  Work  and  Contract  Data  Requirements  List  [2]. 
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Figure  6.  DOD-STD-2167  Products  and  Reviews  as  Tailored  for  CCPDS-R 


SECTION  S 


FORMAL  CONDUCT  OF  THE  SEE 


With  the  completion  of  the  Ml'l'kE  dry  nm  of  the  SEE  and  die  associated  SEE  exercise 
specification,  ground  niles,  and  evaluation  criteria,  the  Government  was  fully  prepared  to  conduct  the 
SEE  as  part  of  the  CCPDS-R  FSD/P  source  selecdoa  This  section  describes  how  the  Government 
conducted  the  SEE  relative  to  the  plans  described  in  secticm  3.  It  details  how  the  Government  released 
the  SEE  to  the  offerors,  the  products  delivered  by  the  offerors,  the  Government’s  evaluation  team,  the 
tools  and  techniques  the  team  used  to  aid  in  the  evaluation  of  the  SEE  products,  and  the  Government's 
overall  approach  for  evaluation  of  the  offerors'  SEE  products. 


5.1  ISSUANCE  TO  THE  OFFERORS 

The  SEE  was  issued  to  the  offerors  following  the  plan  described  in  section  3.4.  The  offerors 
received  a  copy  of  the  SEE  section  M  evaluation  criteria  and  a  preliminary  set  of  SEE  instructions  to  the 
offeror  in  the  request  for  proposal  package,  issued  on  10  October  1986.  Upon  submission  of  the 
offerors'  proposals  to  the  Government  on  10  November  1986,  the  offerors  received  the  SEE  detailed 
instructions  to  the  offeror  and  the  SEE  system  specifications.  The  offerors  were  then  given  3  1/2  weeks 
to  deliver  their  SEE  products,  due  on  3  December  1986.  For  each  offeror,  the  Goverrunent  spent 
approximately  four  days  evaluating  the  delivered  SEE  products,  one  day  conducting  an  audit  at  the 
offeror's  facility,  and  two  days  finalizing  the  evaluation  results. 

This  method  of  issuing  the  SEE  to  the  offerors  worked  out  beneficially.  Rrst,  the  offerors 
benefitted  by  tx)t  having  to  write  their  proposals  arid  develop  the  SEE  products  at  the  same  time. 

Second,  it  allowed  the  Government  SEE  evaluation  team  time  during  the  proposal  evaluation  period  to 
review  each  offeror's  SDP  prior  to  receiving  the  SEE  products.  (The  SDP  defines  the  offeror's 
software  engineering  approach  --  methodology,  tool  set,  ADL,  and  termirwlogy  -  and  is  the  baseline 
against  which  the  offeror’s  products  were  to  be  evaluated.)  If  the  Government  had  to  review  each 
offeror’s  SDP  and  the  SEE  products  at  the  same  time,  either  the  Government  would  have  required  a 
longer  time  period  to  review  the  SEE  products  (as  opposed  to  4  days  per  offeror)  or  the  staff-hours 
required  of  the  SEE  evaluation  team  would  have  been  overwhelming. 


5.2  PRODUCTS  RECEIVED 

As  discussed  in  section  2.12.5,  the  Government  expected  the  offerors  to  submit  all 
requirements  analysis  and  design  products,  both  textual  and  graphic,  which  the  offerors  generated  as 
part  of  their  meth^ology  and  which  are  required  by  DOD-STD-2167,  as  tailored  for  CCPDS-R  (e.g., 
software  requirements  specifications,  software  top-level  design  documents,  software  detailed  design 
documents  (SDDDs),  performance  analyses,  etc.);  all  textual  products  the  offerors  generated  (e.g., 
requirements  analysis  conclusions  and  documentation,  Ada  design  language  listings,  etc.)  in  both 
^’ardcopy  form  and  in  machine-readable,  9-track  tape;  and  a  briefing  to  the  Government  on  the  offerors' 
SEE  results. 
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In  general,  the  Government  did  receive  most  of  the  expected  products  from  the  offerors.  In 
some  cases,  the  Government  received  documentation  that  was  not  required  (e.g.,  diaries  of  the  entire 
SEE  effort).  However,  the  Government  did  not  receive  all  of  the  expected  "inteimediate"  products 
(e.g.,  data  flow  diagrams),  leading  the  Government  to  ccMtclude  that  in  the  future  the  instructions  may 
need  to  be  clarified  to  ensure  that  the  offerors  are  aware  that  the  "intermediate"  products  are  required. 
Overall,  the  SEE  products  delivered  were  of  sufficient  quality,  content  and  scope  to  conduct  a  thorough 
analysis  of  the  offeror's  software  engineering  capabilities. 


5.3  GOVERNMENT  EVALUATION  TEAM 

The  Government  SEE  evaluation  team  consisted  of  die  Source  Selectiot.  Evaluation  Board 
(SSEB)  and  eight  technical  advisors.  The  evaluation  team  was  broken  down  into  tl  :?e  groups  based  on 
the  three  factors  (methodologies,  design,  and  team  expertise)  that  comprised  the  SEE  technical 
evaluation  item.  Each  member  was  assigned  to  one  and  only  one  group,  and  each  group  was 
responsible  for  evaluating  the  offerors'  SEE  products  for  only  that  particular  assigned  factor  and 
associated  standards.  This  division  of  labor  greaUy  expedited  the  process  of  evaluating  the  SEE 
products,  since  it  reduced  the  amount  of  material  any  one  individual  needed  to  evaluate  and,  more 
importantly,  it  reduced  the  amount  of  evaluation  assessment  documentation  that  any  one  individual 
ne^ed  to  prepare.  Although  the  SEE  evaluation  team  members  were  assigned  to  only  one  group,  all 
evaluators,  regardless  of  the  group  to  which  they  were  assigned,  were  permitted  and  encouraged  to 
provide  inputs  to  any  of  the  three  groups/factors.  Freqirent  interaction  did  in  fact  occur  among  the 
different  group  members  during  the  evaluation  process  and  resulted  in  an  effective  and  rapid  interchange 
of  technical  evaluation  assessments. 


5.4  EVALUATION  TOOLS  AND  TECHNIQUES 

As  stated  in  section  3.3,  the  Government  planned  to  use  the  Electronic  Systems  Division 
acquisition  suppon  envirotunent  and  a  set  of  checklist  questions  to  assist  in  the  evaluation  of  the 
offerors'  SEE  products. 

5.4.1  EASE 

The  Government  intended  to  use  the  EASE  tool  for  browsing  through  the  offerors'  textual 
products  and  for  assisting  in  the  evaluation  of  the  ADL  submitted  with  the  SEE  products.  However, 
due  to  the  limited  EASE  funaionality  and  logistical  problems,  the  Government  used  the  EASE  tool  only 
for  verifying  that  an  offeror's  ADL  was  compilable. 

5.4.2  Checklist  Questions 

Although  prior  to  the  actual  source  selection  evaluation  the  Government  thought  that  checklist 
questions,  generate  by  the  MITRE  dry-run  team,  would  be  used  by  the  evaluators  as  they  were  going 
through  the  material  to  determine  if  factors  had  actuaUy  been  met,  it  was  found  that  this  was  not  what 
happened  during  the  evaluation.  This  change  in  the  use  of  the  checklist  questions  from  what  was 
originally  intended  was  made  because  the  checklist  qirestions  were  found  to  be  too  general.  It  was  not 
until  the  Government  team  saw  the  offerors'  actual  SEE  products  that  any  specific  questions  could  be 
generated.  The  checklist  questions  were  thus  fourxl  to  be  of  minimal  benefft,  being  used  only  by  those 
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evaluators  who  had  not  been  part  of  the  original  MITRE  diy-nm  team  as  a  means  of  coming  to  ^)eed 
(XI  the  type  of  details  die  Government  was  looking  for,  and  by  the  e^rimced  team  members  simply  as 
reminders.  They  were  not  used  as  a  means  to  detennine  whether  or  not  factors  and  standards  had  b^n 
met,  or  as  the  b^is  for  the  questions  to  be  asked  during  the  in-house  visit 


5.5  GOVERNMENT  EVALUATION  APPROACH 

As  discussed  in  section  3.5,  upcxi  receipt  of  the  offerors'  SEE  products,  the  Government 
intended  to  perfoim  a  first-pass  evaluation  of  each  offeror’s  SEE  products,  lasting  approximately  one- 
week  per  offeror,  using  each  offeror's  proposed  CCPDS-R  FSD^  SDP  as  the  defmition  of  the 
offeror's  software  engineering  methodology,  conduct  a  1-day  audit  at  each  offeror's  facility;  and,  using 
the  first-pass  evaluation  and  the  results  of  the  audit  to  produce  a  final  evaluation  of  each  offeror's  SEE 
products  within  one  week  of  the  audit. 

5.5.1  First-Pass  Evaluation 

The  Government  evaluated  the  ofterors'  SEE  products  against  the  prepared  factors  and 
standards.  As  a  basis  for  this  evaluation,  the  Government  used  each  offeror's  SDP  (evaluated  during 
source  selection  prior  to  receiving  the  SEE  products),  along  with  any  augmentations  to  it,  to  determine 
whether  or  not  the  offeror's  methcxlologies,  as  described  in  the  SDP,  were  followed  during  the 
development  of  the  SEE  products.  The  Govemmertt  evaluation  was  to  determine  not  only  that  eac^ 
offeror's  SDP  was  followed  in  the  development  of  the  SEE  products,  but  that  the  requirements  analysis 
and  design  methodologies  defined  by  the  SDPs  were  adequate. 

As  strengths  and  weaknesses  in  an  offeror's  SEE  products  or  SDP  were  identified,  vis-a-vis 
the  factors  and  standards,  the  evaluators  documented  them.  For  those  instances  where  the  evaluators 
could  not  find  the  information  necessary  to  evaluate  a  starxlard,  were  not  sure  of  the  offeror's 
motivation  or  rationale,  had  any  (juestions  about  the  products,  or  where  the  evaluation  information 
could  not  be  ascertained  directly  from  the  SEE  products  delivered,  the  evaluators  prepared  questions  to 
be  asked  during  the  in-house  audit  In  addition,  the  Government  evaluation  team  generated  questicxis  to 
verify  its  own  evaluation  of  whether  or  not  system  requirements  had  been  met 

Althou^  it  was  originally  planned  that  one  list  of  questions  would  be  prepared  for  each 
offeror,  and  would  be  presented  to  the  offeror  during  the  question  and  answer  period  of  the  in-house 
audit  the  Government  concluded  during  the  first-pass  evaluation  that  the  (}uestions  for  each  offeror  fell 
into  two  categories:  those  the  Government  would  best  benefit  from  by  allowing  the  offeror  24  hours 
during  which  to  prepare  an  answer  and  those  for  which  the  Government  would  best  benefit  from  by  not 
allowing  the  offeror  mote  than  5  minutes  during  which  to  prepare  an  answer.  Therefore,  during  the 
first-pass  evaluation,  the  Government  prepared  two  sets  of  questions  for  each  offeror  based  on  the 
results  of  the  Government's  evaluation  of  the  following  SEE  factors. 

5.5.1.1  Methodology  Factor 

Each  offeror’s  requirements  analysis  atKl  design  methodologies  were  evaluated  to  ensure  that 
they  adequately  addressed  the  major  issues  in  each  phase  and  that  the  methodologies  were  compatible. 
This  was  accomplished  by  reviewing  the  SEE  products  to  determine  if  the  methodologies  were  robust 
and  cohesive  and  to  ensure  that  the  methodologies  were  consistent  with  each  other  and  provided  a 
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distinction  between  the  end  of  requirements  analysis  and  the  beginning  of  design.  A  methodology  was 
considered  robust  if  it  adequately  and  completely  addressed  modem  software  engineering  issues  for  a 
real-time  system.  Attributes  of  the  SEE  products  that  contribute  to  requirements  analysis  methodology 
robustness  include,  but  are  not  limited  to,  iitclusion  of  performance  aiialysis  during  requirements 
analysis,  detection,  and  resoluticm  of  specification  ambiguities  known  to  exist  in  the  SEE  system 
qtecification,  effective  employment  of  measures  for  tracing  requirements,  and  identification  of  derived 
requirements.  In  addition,  the  SEE  products  were  reviewed  to  ensure  that  the  SEE  products  contained 
acceptable  ADL  and  graphical  representations  consistent  throughout  the  SEE  products.  Questions 
generated  for  the  in-house  audit  were  intended  to  clarify  methodology  questions. 

5.5.1.2  Design  Factor 

Each  offeror’s  design  was  evaluated  to  ensure  that  it  addressed  all  of  the  functional  and  quality 
requirements  contained  in  the  SEE  system  specification,  as  appropriate,  for  the  subset  of  the  architecture 
which  the  offeror  elected  to  design;  Aat  it  demonstrated  sufficient  stmcture  to  support  modularity, 
flexibility,  and  ease  of  change  and  growth;  and  that  it  demonstrated  no  deadlock  or  race  conditions.  The 
design  was  also  evaluated  to  ensure  that  it  employed  effective  approaches  for  managing  data  and  control 
flows;  for  handling  initialization,  termination,  and  exceptions;  ai^  for  meeting  performance  aixl  capacity 
requirements.  For  this  factor,  the  evaluation  team  generated  questions  for  the  in-house  audit  which 
were  intended  to  clarify  design  questions  and  to  verify  the  Government's  evaluation  of  whether  or  not 
certain  SEE  system  specification  requirements  were  met 

5J.1.3  Team  Expertise  Factor 

Based  on  the  results  of  the  exercise,  the  Government  evaluated  the  offerors  on  their  compliance 
with  the  SDP,  on  the  knowledge  demonstrated  by  the  offerors'  SEE  team  members  of  their  SDP 
policies  and  procedures  and  their  tools,  and  on  the  offerors'  proper  use  of  Ada  as  a  design  language  to 
represent  the  design.  The  offerors  were  also  evaluated  to  assess  whether  their  teams  consisted  of 
individuals  well  versed  in  software  engineering  and  real-time  applications,  and  whether  the  teams 
included  strong  software  engineering/Ada  technical  leaders.  The  Government  evaluated  the  offerors  on 
their  teams'  knowledge  of  the  SDP  policies,  tools,  software  engineering,  and  real-time  api^ications  via 
the  responses  to  the  "five-minute,"  spontaneous  questions  posed  to  the  offerors  during  the  in-house 
audit.  By  addressing  many  of  the  spontaneous  questirms  to  particular  offeror  SEE  team  members,  the 
Government  was  able  to  deteimine  the  knowledge  of  an  offeror’s  entire  SEE  team  and  not  just  that  of 
those  members  whom  the  offeror  chose  to  have  respond. 

5.5.2  Audit 

Following  the  first-pass  evaluations,  the  Government  conducted  a  1-day  audit  at  each  offeror’s 
facility.  As  discussed  in  section  3.5.2,  the  purpose  of  the  Government  audit  was  to  verify  the 
Government's  first-pass  evaluation  and  to  obtain  additional  information  necessary  to  complete  the  SEE 
evaluation.  The  offerors  were  required  to  prepare  a  briefing  of  their  SEE  conclusions,  and  to  answer 
the  Government's  questions.  The  offerors  were  evaluated  during  the  in-house  audit  based  on  their 
ability  to  provide  die  required  information  during  the  in-house  briefing  and  adequate  answers  during  the 
question  and  answer  session. 

In  accordance  with  the  procedures  defined  in  section  5.5.1,  the  Government  submitted  two  sets 
of  questions  to  the  offerors  for  the  in-house  audit  One  set  of  questions  was  submitted  24  hours  in 
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advance,  for  which  the  offerors  were  required  to  respond  during  the  audit  as  well  as  to  provide  formal, 
written  responses  to  the  Government  The  other  set  of  questicms  was  given  to  the  offerors  5  minutes  in 
advance,  for  which  the  offerors  were  required  to  respcmd  immediately  and  for  which  the  Government 
maintained  a  record  via  cassette  tape. 

SS3  Evaluation  Completion 

Following  the  in-house  audit  at  each  offeror's  facility,  the  Government  easily  completed  its 
evaluation  of  the  offerors'  SEE  products  within  a  week  of  the  in-house  audit  The  comf^etirai  consisted 
of  updating  the  first-pass  evaluation  assessments  and  associated  idoitification  of  strengths  and 
weaknesses  to  reflect  the  additional,  clarifying  information  obtained  from  the  in-house  audit  In 
addition,  the  Government  transcribed  the  cassette-recorded  responses  to  the  spontaneous  "five-minute" 
questions.  The  transcripts,  together  with  the  formal  responses  to  the  "twenty-four  hour"  questions 
were  then  entered  into  ^e  offerors'  official  submission  of  SEE  products. 
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SECTION  6 


SOURCE  SELECTION  LESSONS  LEARNED 


As  a  result  of  conducting  the  SEE  during  the  CCPDS-R  FSD/P  source  selection,  the 
Government  identified  a  number  of  lessons  learned  concerning  the  administiatiai  of  a  software 
engineering  exercise.  These  lessons  learned  relate  to  deliverable  products,  exercise  scope  and  duration. 
Government  evaluation  tools  and  techniques,  and  Government  evaluation  approach.  The  lessons 
learned,  presented  herein,  are  intended  to  describe  not  only  how  the  SEE  mi^t  be  changed  for  future 
use  or  what  did  not  woric  out  as  well  as  possible,  but  also  to  discuss  those  aspects  of  the  SEE  that  did 
work  well  and  should  be  repeated  in  the  future. 


6.1  DELIVERABLE  PRODUCTS 

In  some  cases,  offerors  did  not  submit  all  "intermediate"  SEE  products  (e.g.,  data  flow 
diagrams)  which  were  expected  by  the  Government.  Therefore,  future  i^ngrams  which  elect  to  carry 
out  a  SEE  may  need  to  evaluate  their  instructions  to  the  offerors  to  see  if  they  must  be  clarified  to  ensure 
that  the  offerors  are  aware  that  all  requirements  analysis  and  design  products,  itKluding  "intermediate" 
products,  are  deliverable  to  the  Government 


6.2  EXERCISE  SCOPE  AND  DURATION 

As  a  result  of  the  use  of  the  SEE  during  source  selection,  the  Government  concluded  that  the 
time,  level,  and  the  coverage  of  the  SEE  was  adequate.  The  volume  and  depth  of  the  offerors'  delivered 
SEE  products  indicate  that  3  1/2  weeks  was  sufficient  time. 

It  was  apparent  during  the  evaluation  of  the  offerors'  performance  on  the  SEE  that  it  did 
provide  the  Government  with  the  answers  it  was  looking  for  concerning  the  offerors'  ability  to 
assemble  a  SEE  team  and  address  software  engineering  and  Ada  issues,  in  the  context  of  the  offerors' 
SDP.  The  level  of  requirements  in  the  system  specification  provided  the  opportunity  for  the  offerors  to 
demonstrate  their  ability  in  the  pertinent  areas  (e.g.,  real-time  system  design,  modem  software 
engineering  practices,  Ada).  It  is  not  felt  that  a  more  difficult  set  of  requirements  would  have  added 
anything  to  the  Government's  knowledge  of  the  offerors'  aUlity.  To  have  increased  the  coverage  of  the 
SEE  requirements,  or  to  have  broadened  the  system,  could  have  had  the  negative  impact  of  forcing  the 
offerors  to  cover  more  area  with  less  depth.  The  Government  feels  that  no  significant  amount  of  new 
evaluation  information  would  have  been  gained,  had  more  time  been  allocated  to  the  offerors  for 
completing  the  exercise.  More  ACtt.  might  have  been  generated,  or  the  products  might  have  been  more 
complete,  but  it  would  not  have  added  anything  to  the  Government's  assessment  of  the  offerors'  ability 
to  perform  requirements  analysis  and  design  a  real-time  system. 
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6.3  EVALUATION  TOOLS  AND  TECHNIQUES 

As  mentioned  in  section  S.4,  the  Govemmera  used  the  ESD  acquisition  support  environment 
and  a  set  of  checklist  questions  to  assist  in  its  evaluation  of  the  offerors'  SEE  products.  In  addition,  the 
Government  relied  heavily  on  word  processors  to  expedite  its  evaluation  and  associated  documentation 
efforts.  The  Government  made  the  following  observations  regarding  the  use  of  these  tools  during  die 
evaluation  of  the  offerors'  SEE  products. 

6.3.1  EASE 

As  discussed  in  section  S.4,  EASE  use  was  attonpted  and  largely  abandoned  during  source 
selection.  The  Government  had  trained  a  large  portion  of  the  evaluation  team  in  the  use  of  EASE; 
however,  the  investment  was  not  worth  the  return  due  to  die  limited  EASE  functionality  and  the 
logistical  proUems  associated  with  using  a  computer  facility  remote  from  the  source  selection.  The 
team  members  also  felt  that  EASE  was  not  really  essential,  given  the  volume  of  SEE  materials 
submitted. 

At  the  time  of  the  CCPDS-R  FSD/P  source  selection,  EASE  provided  text  editing  and  Ada 
compilation  functions,  but  did  not  provide  tools  to  assist  in  identifying  control  flows  and  data  flows, 
perform  syntax-related  browsing  and  cross-referencing,  or  assess  compliance  to  coding/design 
standards.  It  required  significant  manual  overhead  for  such  activities  as  loading  tapes  and  providing 
backups,  and  because  EASE  was  not  collocated  with  the  source  selection  facility,  there  was  time- 
consuming  travel  to  transport  materials  to  the  EASE  facility  for  evaluation.  The  EASE  facility  also  had 
to  be  locked  and  other  EASE  users  could  not  have  access  while  source  selection  sensitive  materials  were 
installed.  Not  until  these  largely  logistical  deficierKies  can  be  overcome  will  EASE  and  similar  tools 
become  useful  tools  for  SEE  evaluations.  Therefore,  before  using  EASE  or  similar  tools  on  future 
software  engineering  exercises,  programs  should  first  assess  the  functionality,  ease-of-use,  logistics, 
and  potential  benefits.  If  the  selected  tool  is  deficiem  in  any  of  these  areas,  then  its  exact  use  during  the 
evaluation  should  be  clearly  specified  prior  to  source  selectioa  If  programs  opt  to  use  automated  tools 
in  the  future,  regardless  of  whether  or  not  any  of  these  deficiencies  still  exist,  these  programs  should 
consider  training  fewer  evaluation  team  monbers  since  the  cost  and  time  required  to  uiKlergo  such 
gaining  is  likely  to  be  significam. 

6.3.2  Checklist  Questions 

As  discussed  in  section  5.4.2,  the  Government  concluded  that  the  checklist  questions  prepared 
prior  to  source  selection  did  not  prove  as  useful  as  had  been  anticipated  and  were  not  worth  the  amount 
of  time  it  took  to  prepare  them.  As  an  evaluation  tool,  the  questions  were  not  very  beneficial  arxl 
serious  consideration  should  be  given  to  either  not  using  them  or  not  investing  so  much  time  in 
preparing  them. 

6J.3  Word  Processing  Capabilities 

At  the  start  of  the  evaluation  of  the  offerors'  SEE  products,  the  Government  SEE  evaluation 
team  had  only  minimal  word-processing  capabilities.  As  the  evaluation  continued,  more  word- 
processing  capability  was  secured,  arxl  though  it  was  helpful,  it  was  still  not  at  an  adequate  level.  For 
the  CCPDS-R  FSD/P  source  selection,  the  SEE  evaluation  would  have  been  expedited  if  there  had  been 
a  separate  word  processor  for  each  of  the  three  groups  which  made  up  the  team.  A  laser  printer  capable 
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of  producing  letter-quality  text  and  viewgraphs  is  also  necessary.  In  die  future,  programs  conducting  a 
SJ^  stwuld  ensure  that  sufficient  word-processing  capability  is  provided  so  that  there  is  no  contention 
of  resources  when  documenting  the  SEE  evaluation  results  against  die  factors  and  standards. 


6.4  IN-HOUSE  AUDIT 

As  a  result  of  conducting  the  in-house  audit,  the  Government  concluded  that  a  1  -day  audit  at  the 
offerors'  facilities  was  both  beneficial  and  of  sufficient  time.  Future  programs  which  institute  software 
oigineering  exercises  are  strongly  encouraged  to  conduct  such  audits  if  time  and  logistics  permit 
Furthermore,  as  a  result  of  conducting  the  in-house  audit  the  Government  made  several  observations 
regarding  the  submission  of  detailed  questions  to  the  offerors  and  the  maintenance  of  a  transcript  of 
offeror  responses. 

6.4.1  Detailed  Questions 

As  discussed  in  section  5.5. 1 ,  the  Government  altered  its  method  for  presenting  detailed 
questions  to  the  offerors  as  a  result  of  the  first-pass  evaluation.  The  revised  approach,  consisting  of 
two  sets  of  questions,  one  submitted  24  hours  in  advance  and  one  5  minutes  in  advance,  proved 
successful;  its  use  is  therefore  recommended  for  other  programs  that  may  conduct  a  SEE  audit  In  the 
case  of  the  24-hour  set  the  Government  was  aUe  to  get  answers  to  questions  that  the  offerors  could  not 
have  answered  as  completely  or  in  as  much  detail  if  they  had  not  had  some  time  to  prepare.  For  the  5- 
minute  set,  the  Government  was  able  to  evaluate  the  offerors  based  on  their  alxlity  to  answer  questions 
extemporaneously  which  should  have  required  no  pneparation  time,  assuming  the  offerors'  teams  were 
fully  trained  in  the  methodologies  as  claimed  in  their  proposals.  The  Government  was  aUe  to  direct 
many  questions  to  particular  offeror  SEE  team  members  based  on  their  area  of  re^xxisibility  on  the 
SEE,  lending  substance  to  the  evaluation  of  the  offerors'  entire  SEE  team.  The  use  of  two  sets  of 
questions  for  the  offerors  provided  discriminating  information  that  could  not  have  been  attained  through 
the  use  of  only  one  set  of  questions. 

6.4.2  SEE  Transcripts 

As  mentioned  in  section  5.5.3,  the  Government  completed  its  evaluation  of  the  offerors'  SEE 
products  by  updating  the  first-pass  evaluations  to  reflect  the  audit  results.  Transcripts  of  the  offerors' 
recorded  spontaneous  audit  responses,  together  with  the  offerors'  formal  responses  to  the  "twenty-four 
hour"  questions  were  entered  into  the  offerors'  official  SEE  product  submissions. 

In  general,  this  approach  to  completing  the  SEE  evaluation  worlced  well.  The  one  major 
drawback  was  the  method  chosen  for  documenting  the  offerors'  responses  to  the  spontaneous 
questions.  Originally,  the  SSEB  (farmed  to  have  the  offerors  maintain  the  written  transcripts  of  these 
responses,  but  this  decision  was  overruled.  Consequently,  the  CCPDS-R  FSD/P  SSEB  had  to 
maintain  the  transcript  However,  transcribing  the  cassette  tapes  (daced  an  overwhelming  burden  on  the 
limited  SSEB  resources  since  it  was  such  an  extremely  tedious,  time-consuming  process.  Therefore,  it 
is  suggested  that  in  the  future,  if  SEE  audits  are  held,  that  either  the  SSEB  be  allowed  to  have  the 
offerors  maintain  the  written  transcripts  or  that  some  alternative  method  be  fouixl,  such  as  only 
requiring  magnetic  recordings  or  videotapes  of  the  responses. 
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SECTION  7 


OFFEROR  FEEDBACK 


In  addition  to  making  its  own  observations  teganling  administering  and  conducting  a  software 
engineering  exercise,  the  Government  solicited  feedback  ftom  the  CCPDS-R  FSD/P  offerors  on  their 
impressions  of  the  SEE.  The  Government  accomplished  this  by  providing  to  the  offerors  an  rational 
questionnaire  at  the  conclusion  of  the  in-house  audit  (see  appendix  D).  In  general,  the  offerors 
responding  to  the  questionnaire  felt  that  the  SEE  was  a  valu^e  exercise.  This  section  summarizes  the 
offeror  feedback,  t^dressing  the  areas  of  the  size  of  the  SEE,  the  exercise  appropriateness,  the 
resources  expended,  and  the  benefits  to  the  offerors. 


7.1  SIZE 

On  the  average,  the  offerors  considered  the  size  of  the  SEE  to  be  of  an  appropriate  level,  both  in 
terms  of  the  time  required  and  the  time  allowed  by  the  Government,  and  in  terms  of  the  SEE  system  that 
the  offerors  were  required  to  design.  The  offerors  made  no  recommendations  to  either  increase  or 
decrease  the  scope  of  the  SEE  or  the  time  allotted  for  it 


7.2  APPROPRIATENESS 

Generally  speaking,  the  offerors  considered  the  SEE  to  be  a  challenging,  appropriate  exercise  in 
relation  to  CCPDS-R  FSD/P  source  selection.  The  offerors  indicated  that  the  focus  of  the  SEE  on 
requirements  analysis  and  design  was  very  appropriate  because  of  the  perceived  high  level  of  risk 
associated  with  the  requirements  analysis  and  design  of  an  Ada  system.  Some  offerors  felt  that  metrics 
should  have  been  included  in  the  SEE,  since  metrics  collection,  reporting,  and  evaluation  are  integral  to 
program  management  Additionally,  some  offerors  felt  that  not  enough  opportunity  was  provided  to 
demonstrate  products  that  they  had  expended  resources  on  (e.g.,  prototypes)  for  use  in  CCPDS-R 
FSD/P. 


7.3  RESOURCES 

On  the  average,  the  offerors  considered  the  resources  expended  on  the  SEE  to  be  of  a 
reasonable  level.  The  percentage  of  time  allotted  by  the  offerors  was  fairly  equally  divided  between 
each  of  the  phases  (i.e.,  requirements  analysis,  top-level  design,  detailed  design,  and  preparation  for 
arxl  participation  in  the  in-house  audit).  The  average  amount  of  resources  expended  by  the  offerors  was 
a  little  less  than  1  staff-year. 


7.4  BENEFITS 

Overall,  the  offerors  assessed  the  SEE  as  beneficial,  for  several  reasons.  First,  the  SEE 
provided  the  offerors  an  opportunity  to  exercise  and  refine  their  software  engineering  methodologies. 
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Prior  to  the  SEE,  the  requirements  analysis  and  design  methodologies  included  in  the  offerors'  SDPs 
had  not  been  fully  utiliz^.  The  SEE  provided  an  opportunity  for  the  offerors  to  exercise  their 
methodologies  on  an  actual,  albeit  small,  program  and  to  receive  feedback  internal  to  the  offerors' 
organizations  on  those  methodologies  and  the  products  delivered  as  a  result  of  utilizing  them.  This 
feedback  provided  the  offerors  with  an  opportunity  to  refine  their  methodologies  where  necessary,  prior 
to  utilizing  them  on  a  large  program  such  as  CCPDS-R.  Second,  the  SEE  provided  an  actual  illustration 
of  the  benefits  of  various  software  engineering  aj^xoaches  (e.g.,  prototyping,  reusable  components, 
etc.).  This  provided  offeror  insight  into  the  value  of  these  approaches  an^or  the  need  to  modify  these 
software  engineering  approaches  for  CCPDS-R  FSD/P.  Third,  the  SEE  provided  CCPDS-R  related 
experience  which  can  then  be  applied  to  the  CCPDS-R  FSD/P  phase.  Finally,  successftiUy 
accomplidiing  the  SEE  using  their  chosen  methodologies  and  Ada  provided  the  offerors  with  increased 
confidence  in  those  methodologies,  their  Ada  expertise,  and  their  software  development  core  team. 
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SECTION  8 


CONCLUSIONS/RECOMMENDATIONS 


Overall,  both  the  SEE  dry  run  and  the  incoiporation  of  the  SEE  into  the  FSD/P  source  selection 
were  successful.  This  section  summarizes  the  Government's  conclusions  and  recommendations 
resulting  fiom  this  successful  CCPDS-R  SEE,  first  firom  the  perspective  of  the  dry  run  of  the  SEE  and 
then  from  the  actual  use  of  the  SEE  during  source  selectioa 


8.1  DRY  RUN 

The  Government  assessed  the  dry  run  of  the  SEE  as  extremely  beneficial,  given  the  lessons 
learned  from  that  effort.  More  importandy,  however,  based  on  the  results  of  the  dry  run,  the 
Government  determined  that  the  software  engineering  exercise  demonstrated  strong  potential  for  being 
an  effective  and  discriminating  source  selection  technique.  This  section  surrunarizes  the  Government's 
pre-source  selection  SEE  conclusions  and  recommendations  relative  to  the  dry-mn  objectives,  software 
engineering,  and  Ada. 

8.1.1  Objectives 

As  stated  in  section  2,  the  primary  objectives  of  the  MITRE  SEE  dry  run  were  to  generate  a 
clearly  defined  SEE  system  specification,  to  develop  the  ground  rules  for  the  offerors  to  follow  when 
conducting  the  SEE,  to  identify  a  discriminating  set  of  evaluation  criteria,  and  to  assess  whether  the 
SEE  could  reasonably  be  done  in  the  time  allotted  to  the  offerors.  The  secondary  objective  was  to 
educate  staff  in  software  methodologies,  Ada,  ADL  and  DOD-STD-2167.  The  SEE  dry  run  achieved 
all  these  objectives  satisfactorily.  As  a  result  of  dry  running  the  SEE,  the  Government  was  able  to 

a.  Analyze  the  draft  SEE  system  specification  thoroughly,  identify  weaknesses  in  the  draft 
specification,  resolve  these  weaknesses,  and  generate  a  final,  coiKise  SEE  system 
specification  that  contained  heretofore  omitted  requirements  pertinent  to  CXPDS-R,  that 
would  serve  as  key  technical  discriminators 

b .  Develop  a  set  of  offeror  instructions  by  which  the  Government  scoped  the  SEE,  expedited 
both  the  offeror  preparation  effort  arai  the  Government  evaluation,  and  maintained  the 
fairness  and  objectivity  of  the  SEE  effort 

c.  Identify  a  low  level  set  of  technical  discriminators  geared  specifically  to  the  SEE  system 
specification  and  Ada,  which  the  Government  felt  would  enable  it  to  separate  form  from 
substance  in  the  offerors'  results  and  thus  distinguish  those  offerors  who  have  strong 
software  engineering/Ada  capabilities  from  those  who  do  not 

d .  Verify  that  the  SEE,  as  scoped  per  the  detailed  offeror  instructions,  could  reasonably  be 
done  within  the  3  1/2  weeks  allotted  to  the  offerors 
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e.  Gain  fiiither  knowledge,  depending  on  the  skill  of  the  individual  SEE  team  member,  in 
requirements  analysis  and  design  methodologies,  Ada,  ADL,  DOD-STD-2167,  all  of  which 
would  prove  useful  to  the  CCPDS-R  program  office  during  both  the  source  selection  and 
the  FSD/P  phase. 

MITRE  expended  approximately  fifteen  staff  months  of  effort  dry  running  the  SEE,  from  initial 
conceptualization  of  the  SEE  to  completion  of  all  SEE  RFP  documentatioa  Thus,  the  Government 
considered  the  SEE  dry-run  effort  somewhat  costly;  however,  the  Government  considered  that  the 
benefits  far  outweighed  the  costs.  Since  the  SEE  was  a  new  source  selection  technique  at  ESD,  the 
Government  considered  the  dry  running  of  the  SEE  mandatory  to  test  out  the  ccmcept  of  the  SEE,  to 
verify  that  the  SEE  was  a  reasonable  and  work^le  source  selection  techniqtie,  and  to  ensure  the  overall 
success  of  the  SEE  as  a  source  selection  technique  for  CCPDS-R  and  other  future  programs. 

Moreover,  by  dry  nuining  the  SEE,  the  Government  was  better  able  to  identify  specification 
requirements  and  evaluation  criteria  it  felt  would  serve  as  true  discriminators  during  source  selection  and 
to  train  staff  in  software  engineering  methodologies  and  Ada  for  both  present  and  future  use.  Given 
these  benefits,  it  is  strongly  recommended  that  when  a  program  includes  a  software  engineering 
exercise  as  part  of  its  source  selection  approach,  it  dry  run  the  exercise  to  some  extent  before  the  release 
of  the  exercise  to  offerors.  As  a  minimum,  the  dry  run  should  focus  on  the  generation  of  an  appropriate 
system  specification  and  on  the  development  of  discriminating  evaluation  criteria. 

8.1.2  Software  Engineering  and  Ada 

A  number  of  software  engineering  and  Ada  lessons  learned  resulted  from  the  MITRE  SEE  dry 
run.  The  major  conclusions  and  associated  reconunendations  are  as  follows: 

a.  More  time  is  needed  for  requirements  analysis  than  is  traditionally  allocated.  This  extra 
time  is  due  to  the  difficulty  of  correctly  interpreting  user  requirements  and  intentions,  as 
well  as  the  increased  level  of  detail  required  by  DOD-STD-2167.  Programs  with  a  large 
software  development  component  should  therefore  plan  appropriately  for  this  additional 
time. 

b .  Daily/weekly  Government  participation  during  the  contractor’s  requirements  analysis  effort 
may  facilitate  progress  during  that  phase  by  helping  to  clarify  specification  ambiguities  and 
prevent  incorrect  assumptions  by  the  contractor.  It  is  therefore  suggested  that  software 
acquisition  programs  consider  having  Government  representatives  on  site  during  the 
contractor's  requirements  analysis  effort  to  expedite  those  activities. 

c.  Object  oriented  design  as  defined  in  Booch’s  "Software  Engineering  with  Ada"  provides 
useful  guidelines  but  does  not  constitute  a  complete  methodology.  Consequently, 
programs  using  Booch's  OOD  should  either  augment  it  to  overcome  the  shortfalis  in  it  or 
should  seek  other  alternative  methodologies.  Alternative  methodologies  should  also  be 
scrutinized  for  completeness. 

d.  DOD-STD-2167  appears  to  require  specification  and  authentication  of  data  (e.g.,  timing 
and  sizing  information)  which  seems  premature  and  in  some  cases  impossible  to  answer  for 
the  given  stage  of  development  This  problem  seems  especially  true  for  Ada  developments 
for  which  no  previous  data  is  available  upon  which  decisions  and  estimates  can  be  made. 
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Therefore,  programs  using  DOD'STD-2167  should  consider  tailoring  the  standard  so  that 
specification  and  authendcation  of  data  occurs  at  achievaUe  and  realistic  milestimes. 


e.  With  the  Ada  language  and  its  tasking  construct,  applications  software,  and  not  just 
operating  system  software,  must  consider  and  han^e  control  flow  issues  such  as  deadlock 
and  process  starvation.  Thus,  {xograms  using  Ada  should  ensure  that  the  software 
development  methodologies  emjdoyed  on  the  program  include  techniques  for  designing 
effective  controls  for  the  detection  andA)r  preventitm  of  deadlock  and  process  starvaticm. 

f .  An  Ada-based  design  language  by  itself  is  not  a  sufficient  tool  for  effecting  clarity  and 
communication  of  global  system  design  infonnation  either  among  developers  or  between 
developers  and  the  Government  Hence,  programs  using  ADL  should  require  that  a 
graphical  design  representation  technique,  ctmsistent  with  the  ADL,  also  be  used  for 
portraying  design  information. 

g .  Availability  and  retention  of  qualified  Ada  engineers  constitutes  a  high  risk  on  Ada 
developments.  Consequently,  programs  using  Ada  should  investigate  the  use  of  different 
contracting  vehicles  and  incentives  to  obtain  and  retain  qualified  Ada  engineers  both  within 
the  Government  agencies  and  the  contractors'  orgaiuzations. 

h .  For  programs  using  Ada,  Ada  training  must  occur  at  all  levels  of  the  software  development 
aixl  acquisition  teams.  Proper  training  in  Ada,  however,  takes  Icmger  than  for  other 
languages.  Therefore,  programs  designing  and/or  implementing  in  Ada  should  require 
extensive  Ada  training  for  both  Government  and  contractor  personnel,  as  appropriate,  and 
should  plan  and  account  for  any  additional  time  and  effort  required  to  do  so. 

The  Government  also  concluded  that  a  software  engineering  exercise  serves  as  an  extremely 
effective  vehicle  for  training  perscMviel  in  all  aspects  of  software  acquisition  and  software  engineering. 
The  unique  benefit  of  the  SEE  as  a  training  approach  is  that  it  provides  practical,  interactive,  hands-on 
experience  not  offered  in  typical  non-interactive  theoretical  courses,  and  it  covers  a  range  of  issues, 
such  as  requirements  analysis  methodologies,  design  methodologies,  Ada,  DOE>-STD-2167,  software 
specifications  and  reviews,  and  software  tools  and  technique.  While  the  SEE  is  an  effective  training 
technique,  it  is  costly  to  conduct  since  participants  must  dedicate  significant  amounts  of  time  and  effort 
to  reap  the  benefits.  However,  the  benefits  are  considered  to  far  outweigh  the  cost 


8.2  ACTUAL  SOURCE  SELECTION 

At  the  start  of  the  CCPDS-R  FSD/P  source  selection,  the  Government  cotisidered  the  purpose 
of  the  SEE  to  provide  discriminating  information  that  would  enable  the  Government  to  determine  the 
degree  of  risk  associated  with  each  CCPDS-R  offeror’s  proposed  software  development  methodology 
and  to  determine  the  offeror's  ability  to  organize  a  team  fully  knowledgeable  in  that  methodology  and  in 
Ada,  the  required  CCPDS-R  implementation  language.  This  section  provides  a  summary  of  the 
conclusions  the  Government  reached  regarding  the  SEE  versus  its  CO^DS-R  objectives.  It  also 
provides  some  general  observations  regarding  the  conduct  of  future  software  engineering  exercises. 
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8.2.1  CCPDS-R  SEE  Objectives 

At  the  completion  of  the  FSD/P  source  selectitm,  the  Government  concluded  that  the  CCPDS-R 
SEE  satisfied  its  objectives  resoundingly.  General  conclusions  regarding  the  CCPDS-R  SEE  as  a 
source  selection  technique  may  be  summarized  as  follows: 

a.  The  SEE  was  an  extremely  beneficial  source  selectim  technical  area  evaluation  technique. 
By  having  offerors  develop  actual  products  using  their  proposed  software  development 
approach,  the  SEE  provid^  the  Goverrunent  invaluaUe  insights  as  to  what  an  offeror 
really  can  do  versus  what  an  offeror  claims  he  can  do.  It  provided  a  concrete  example  that 
demonstrated  the  degree  of  robustness  of  an  offeror’s  methodology,  the  offeror’s  ability  to 
follow  the  propos' d  SDP,  and  the  offeror’s  expertise  in  the  proposed  methodology  and 
tool  set.  It  clearly  demonstrated  whether  or  not  an  offeror's  proposed  CCPDS-R  FSD/P 
team  had  sufficient  expertise  to  design  and  develop  a  real-time  system  in  Ada,  as  is  required 
for  CCPDS-R. 

b.  The  SEE  served  as  an  excellent  vehicle  by  which  to  identify  early  problems  in  an  offeror's 
software  approach.  For  example,  the  use  of  the  SEE  helped  to  point  out  incomplete 
methodologies  that  did  not  address  all  of  the  software  engineering  issues,  areas  where  the 
requirements  analysis  and  design  methodologies  conflicted,  and  inadequate  ADL  arxl 
graphical  design  representation  techniques.  By  utxx)vering  these  problems  during  source 
selection,  the  Government  was  better  able  to  focus  on  these  problems  irrunediately  at 
FSD/P  contract  award,  rather  than  waiting  until  they  become  apparent  in  the  development 
phase,  when  problems  are  more  costly  and  difficult  to  correct  and  the  contractor  is  less 
willing  to  make  changes. 

In  addition  to  meeting  its  stated  objectives,  the  SEE  also  provided  some  additional  benefits  not 
originally  anticipated.  In  particular,  the  SEE  assisted  in  the  source  selecticHi  cost  area  evaluation  by 
yielding  valuable  information  on  offeror  capabilities  in  such  areas  as  level  of  experience  with  the 
selected  programming  language  and  tools.  This  additional  insight  into  actual  offeror  capat^ties  enabled 
the  Government  to  generate  more  representative  inputs  for  its  software  cost  estimation  models  and 
thereby  to  assess  cost  aixl  schedule  risk  associated  with  an  offeror’s  software  development  approach  for 
the  CCPDS-R  FSD/P  f^ase.  Also,  as  the  offeror  feedback  indicates,  the  SEE  forced  offerors  to 
solidify  and  test  out  their  methodologies  and  teams  and  thus  to  make  modifications,  as  appropriate,  to 
eliminate  problems  on  their  own  prior  to  the  FSD/P  phase. 

8.2.2  General  Observations 

Given  the  overwhelming  benefits  that  were  reaped  from  the  CCPDS-R  FSD/P  SEE,  the 
Government  SEE  team  strongly  recommends  the  use  of  a  software  engineering  exercise  for  other 
acquisition  programs.  However,  the  team  does  so  with  the  followiiig  caveats: 

a.  To  conduct  a  software  engineering  exercise  is  costly,  both  for  the  Government  and  for  the 
offerors.  For  the  CCPDS-R  SEE,  the  Government  expended  approximately  twenty  staff- 
months  to  dry  run  the  SEE  and  to  evaluate  the  offerors’  SEE  pn^ucts  during  source 
selection.  Offerors  expended  approximately  ten  staff-mtmths  each  to  carry  out  the  exercise. 
As  the  Government  b^mes  more  used  to  conducting  SEEs,  the  level  of  Government 
effort  expended  will  decrease,  perhaps  to  ten  staff-months.  However,  in  any  case,  if  a 
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program  elects  to  conduct  a  software  engineering  exercise,  it  should  be  aware  of  and  aide  to 
acctxnmodate  the  additional  cost 

b .  Evaluation  of  a  software  engineering  exercise  may  add  significant  time  to  a  source  selection 
if  many  offerois  respond  or  if  the  Government  evaluation  team  is  rK>t  well  prepared  in 
advance.  Consequently,  programs  that  opt  to  conduct  a  SEE  should  omsider  rq^roaches 
for  minimizing  the  time  required  to  conduct  and  evaluate  a  SEE.  Possilde  approaches 
include  staggered  release  of  the  exercise.  Government  dry  running  of  the  exercise  as  was 
done  for  CCPE)S-R,  and  strong  technical  and  management  teams  to  evaluate  the  SEE 
products. 

c.  A  software  engineering  exercise  is  of  benefit  to  a  particular  program's  source  selection  only 
if  it  is  tailored  for  that  program.  For  examine,  providing  a  missile  warning  exercise  on  a 
local  area  network  (LAN)  program  would  be  of  little  value  since  the  offerors,  most  likely 
communications  software  engineers,  would  not  be  evaluated  on  those  LAN-unique 
software  areas  which  would  truly  demonstrate  the  offerors'  capabilities  to  implement  the 
real  program.  Also,  a  software  engineering  exercise  is  beneficial  only  if  it  has  specific, 
realistic  goals  in  mind.  For  example,  in  the  case  of  CCPI>S-R,  concern  existed  that  not  all 
offerors  would  be  proficient  in  designing  a  missile  warning,  real-time  system  in  Ada,  as  is 
required  for  CCPDS-R.  The  CCPDS-R  SEE  and  associated  evaluation  criteria  were 
specifically  devised  to  address  this  concern.  Theiefote,  if  a  program  does  choose  to 
conduct  a  software  engineering  exercise,  it  should  ensure  that  the  exercise  is  applicable  to 
the  real  program  and  that  it  is  geared  towards  discerning  particular,  discriminating 
information  about  the  offerors. 

d .  While  a  software  engineering  exercise  provides  discriminating  source  selection 
information,  it  cannot  be  relied  upon  solely  as  a  means  to  select  a  contractor.  For  example, 
situations  may  occur,  such  as  offerors  not  following  the  ground  rules  and  using  resources 
not  proposed  in  the  SDP,  which  may  invalidate  the  SEE  results  and  consequently  its 
use^lness  as  an  evaluation  item.  Thus,  programs  which  elect  to  carry  out  a  software 
^gineering  exercise  should  irKlude  other  evaluation  items  besides  the  exercise  upon  which 
to  make  a  source  selection  decision. 

In  some  respects,  the  CCPDS-R  program  was  fortunate  in  that  it  was  the  first  program  at  ESD 
to  conduct  a  software  engineering  exercise.  Consequently,  industry  was  not  sure  what  to  expect  and, 
therefore,  industry  followed  the  SEE  instructions  completely  arvl  satisfied  the  SEE  intent  fully. 
However,  as  software  engineering  exercises  become  more  common,  the  response  of  industry  may  be  to 
develop  "professional  exercise  teams"  analogous  to  the  specialized  proposal  preparation  teams  now 
evident,  or  to  bring  in  outside  consultants  or  employ  other  similar  vehicles  (e.g.,  submitting  too  much 
material)  which  will  in  essence  circumvent  or  negate  the  intent  of  the  exercise.  Future  programs  that 
choose  to  conduct  software  engineering  exercises  must  be  aware  of  this  possibility  and  thus  take 
additional  precautions  where  necessary  to  prevent  this  situation  from  arising  during  source  selection. 
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APPENDIX  A 


SEE  INSTRUCTIONS  FOR  THE  OFFEROR 
AND  EXERCISE  SPECIFICATION 


This  appendix  contains  the  detailed  instnictions  for  the  offeror  and  the  CCPDS-R  SEE  system 
specification  provided  to  the  offerors  upon  receipt  of  the  CCPDS-R  FSD/P  proposals. 
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CCPDS-R  SOFTWARE  ENGINEERING  EXERCISE 


Detailed  Instructions  for  the  Offeror 


1.0  PURPOSE 

The  purpose  of  the  software  engineering  exercise  (SEE)  is  to  permit  the  Government  to  evaluate 
an  actual  application  of  each  offeror's  software  development  methodology  as  proposed  for  use  during 
the  CCPDS-R  full-scale  development/production  (FSD/P)  phase.  The  Sffi  will  concentrate  exclusively 
on  the  offerors'  approach  to  requirements  analysis,  design,  and  their  interrelationship.  'The 
Government  will  not  evaluate  as  part  of  the  SEE  the  offeror’s  tq)proach  to  imj^ementation,  integration, 
test,  quality  assurance,  configuration  management,  staffing  level,  productivity  measures,  software 
metrics  collection,  and  other  developnent  activities  not  explicitly  mentioned  in  the  following 
paragraphs. 


2.0  INSTRUCTIONS  FOR  THE  OFFERORS 

Each  offeror  will  provide  a  prototypical  example  of  his  proposed  software  development 
approach,  as  applied  to  a  sample  problem  taken  from  ^e  missile  warning  domain.  ['The  attachment], 
"Exercise  Specification,"  presents  the  requirements  for  the  sample  problem.  In  performing  the  exercise, 
the  offeror  shall  comply  with  all  provisions  of  his  proposed  software  development  plan  and  with  section 
3.3  of  the  CCPDS-R  system  specification.  To  the  maximum  extent  practical,  the  offeror  shall  make  use 
of  development  tools  and  procures  that  are  proposed  for  the  CCPDS-R  FSD/P  phase,  as  this  will  be 
viewed  more  favorably  by  the  Government;  deviations  shall  be  noted  by  the  offerors. 

Participation  in  the  exercise  shall  be  limited  to  those  individuals  identified  in  the  offeror’s 
proposal  as  part  of  the  CCPDS-R  full-scale  development  team.  Subcontractors  who  will  be  re.sponsible 
for  software  development  on  CCPDS-R  shall  be  active  participants.  Consultants  shall  be  precluded 
from  participating.  Each  offeror  will  deliver  to  the  Government  all  requested  materials,  in  the  formats 
described  in  section  3,  no  later  than  12  iKX)n  local  time,  3  December  1986.  The  Government  will 
review  this  material  for  a  period  of  time  not  to  exceed  two  (2)  calendar  weeks.  Following  completion  of 
the  Government  review,  a  Government  team  will  conduct  an  (xi-site  visit  at  the  offeror’s  facility,  at 
which  time  the  offeror  shall  brief  his  approach  and  provide  resptxises  to  Govenunent  requests  for 
clarification.  The  Government  will  coordinate  the  schedule  for  the  on-site  visit  with  the  offeror  upon 
receipt  of  the  offeror’s  exercise  results.  Preliminary  plans  are  for  the  Government  to  conduct  the  on-site 
visit  during  the  week  of  15-19  December  1986.  Note  that  there  will  be  no  interaction  between  the 
offeror  and  the  Government  during  the  offeror’s  implementation  of  the  exercise.  Should  the  offeror 
have  any  questions  on  the  exercise,  the  offeror  is  instructed  to  identify  appropriate  assumptions,  to 
document  these  assumptions,  and  proceed  with  the  exercise  based  on  those  assumptions. 

'The  Government  will  conduct  its  evaluation  of  the  offeror's  delivered  materials  and  assess  the 
offeror’s  proposed  methodologies  using  as  a  primary  reference  the  offeror's  Software  Development 
Plan  (SDP)  submitted  with  the  CCPDS-R  proposal,  and  particularly  the  software  standards  a^ 
procedures  contained  within  the  SDP.  The  offeror  may  submit  with  the  SEE  materials  delivered  on 
3  December  1986  an  augmentation  to  the  SDP,  not  to  exceed  fifteen  (15)  pages,  which  provides  further 
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SEE  DETAILED  INSTRUCTIONS  (Continued) 


concise,  technical,  and  explicit  details  regarding  the  offeror's  proposed  software  development  approach 
and  methodologies.  The  Government  will  consider  any  such  augmentation  as  part  of  the  offeror's 
proposal  and  subject  to  Government  evaluation. 

The  Government  will  employ  automated  tools  to  conduct  its  evaluation  of  the  offeror's 
delivered  materials.  Therefore,  as  described  in  section  3,  the  offeror  is  required  to  deliver  some  of  the 
exercise  products  in  machine-readable  format.  In  order  to  assess  the  compatibility  of  the  Government's 
tools  and  the  offeror's  machine-readable  products,  the  offeror  is  requested  to  deliver  to  the  Government 
rx)  later  than  12  noon  local  time,  19  November  1986,  a  demonstration  tape  containing  sample  files  of 
the  offeror’s  methodology  products  (e.g.,  Ada-based  design  language  (ADL)  listings,  etc.)  in  the  same 
format  as  will  be  submitt^  at  the  conclusion  of  the  exercise  period.  The  Government  will  not  evaluate 
the  contents  of  this  demonstration  tape,  but  will  merely  use  the  tape  to  study  artd  resolve  any 
compatibility  issues  that  may  develop  between  the  Government's  tools  and  the  offeror's  tape  output. 
The  sample  files  on  the  demonstration  tape  do  not  need  to  represent  actual  products  of  the  exercise;  they 
need  only  represent  general  products  of  the  offeror's  propws^  methodologies,  the  types  of  which  the 
offeror  will  submit  for  evaluation  at  the  end  of  the  exercise  period. 


3.0  PRODUCTS  OF  THE  EXERCISE 

At  the  conclusion  of  the  exercise  period  on  3  December  1986,  the  offeror  shall  deliver  the 
following  items  to  the  Government  for  evduation: 

a.  A  complete  software  architecture  for  the  sample  problem.  This  ardiitecture  shall  contain  an 
identification  of  software  components,  an  allocation  of  functions  to  these  software 
components,  a  preliminary  specification  of  interfaces,  and  an  indication  of  control  and  data 
flow  throughout  the  system. 

b.  For  two  or  more  offeror-selected  components  of  the  system,  all  requirements  analysis 
conclusions  reached  and  documentatioa  With  respect  to  the  selected  components,  the 
requirements  analysis  shall  represent  a  complete  u^ization  of  the  tools  and  procedures 
proposed  by  the  offeror  for  use  on  CCPDS-R.  The  offeror  shall  identify  any  deviations 
from  these  tools  and  procedures  and  the  associated  rationale  for  these  deviations  in  his 
briefing  to  the  Government 

c.  For  two  or  more  offeror-selected  components  of  the  system,  all  preliminary  design 
documentation,  including  requirements  traceability,  ADL  listings,  and  graphics  products. 
With  respect  to  the  selected  components,  the  preliminary  design  documentation  shall 
represent  a  complete  utilization  of  the  tools  and  procedures  proposed  by  the  offeror  for 
CCPDS-R.  The  offeror  shall  identify  any  deviations  from  these  tools  and  procedures  and 
the  associated  rationale  for  these  deviations  in  his  briefing  to  the  Government 

d .  For  at  least  one  offeror-selected  componoit  of  the  system,  all  detailed  design 
documentation,  including  requirements  traceability,  ADL  listings,  and  graphics  products. 
With  respect  to  the  selected  component(s),  the  detailed  design  documentation  shall 
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represent  a  complete  utilization  of  the  tools  and  procedures  proposed  by  the  offeror  for 
CCPDS-R.  The  offeror  shall  identify  any  deviations  from  Uiese  tools  and  procedures  and 
the  associated  rationale  for  these  deviations  in  his  briefing  to  the  Govemment 

All  textual  products  of  the  exercise,  iiKludii^  requirements  analysis  conclusions  and 
documentation,  ADL  listings,  and  other  design  documentation  shall  be  delivered  to  the  Government 
both  in  hardcopy  form  aixl  in  machine-readable,  9-track:  tape.  Exception  will  be  made  for  materials  that 
the  offeror  does  not  propose  to  create  and/or  maintain  online  during  the  CCPDS-R  FSD/P  contract  In 
particular,  graphical  representations  shall  be  submitted  in  hardcopy  form.  The  tape  shall  be  in  9-track 
1600  bpi  forniat  in  accordance  with  ANSI  X3.27-1978.  ASCII  labelled,  and  with  an  identified  record 
size  and  block  size.  The  block  size  shall  be  512  bytes.  For  readability,  all  tabs  should  be  expanded  to 
spaces.  The  offeror  shall  provide  ten  (10)  copies  of  all  hardcopy  products.  The  products  delivered 
shall  be  clear,  coherent,  legible,  and  prepared  in  sufficient  detail  for  effective  evaluation.  Elaborate 
documentation,  expensive  binding,  detailed  art  work  or  other  embellishments  are  unnecessary.  The 
offeror  shall  include  with  these  products  indices  delineating  the  subject  and  contents  of  the  h^copy 
material  package  and  the  9-track  tape;  the  operating  system  command(s)  used  to  create  the  tape;  a  list  of 
ADL  compilation  units;  and  a  list  of  the  compilation  order  of  these  units. 

In  addition  to  the  delivered  products  described  above,  the  offeror  shall  provide  a  briefing  to  the 
Government  during  the  on-site  visit  that  summarizes  his  experience  in  carrying  out  the  exercise  and 
describes  the  products  generated.  The  briefing  shall  not  exceed  three  (3)  hours  in  duration.  The  topics 
presented  shall  include  the  following: 

I .  Management  approach,  to  itKlude: 

A.  Introduction  of  team  members 

B .  A  description  of  individual  roles  aixl  experieiKe 

II.  An  overview  of  the  requirements  analysis  approach,  to  include: 

A.  A  rationale  for  the  selection  of  the  software  components 

B .  A  description  of  the  tools  and  procedures  employed 

C.  Significant  requirements  is.sues  encountered  and  their  resolution 

D.  A  discussion  of  deviations  from  the  proposed  approach,  and  associated  rationale 

E.  Other  topics  to  be  determined  by  the  offeror 

III.  An  overview  of  the  approach  to  preliminary  and  detailed  design,  to  include: 

A.  A  rationale  for  the  selection  of  the  software  components 

B .  A  description  of  the  tools  and  procedures  employed 

C.  Significant  design  issues  encountered,  alternatives  considered,  and  a  rationale 
for  decisions  made 

D.  A  discussion  of  deviations  from  the  proposed  approach,  and  associated 
rationale 

E.  Other  topics  to  be  determined  by  the  offeror 
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IV.  Other  topics  to  be  detemtined  by  the  offeror 

The  briefing  shall  not  include  any  discussion  of  further  woilc  which  die  offeror  may  have 
completed  following  the  submission  of  the  SEE  products  on  3  December  1986,  since  the  Government 
will  not  evaluate  this  additional  work.  All  paiticipants  in  the  exercise  shall  be  present  at  the  briefing  to 
respond  to  Government  requests  for  clarificatioa  The  offeror  shall  provide  ten  (10)  p^r  copies  of  the 
briefing  slides  and  accompanying  text  at  the  time  of  the  presentatioa  A  transcript  of  the  questiems  and 
answers  will  be  kept.  All  offeror  responses  to  these  Government  clarification  requests  (i.e.,  the 
transcript)  together  with  the  briefing  presentation  mtderial  and  the  products  identified  in  items  (a)-(d) 
above  sh^  be  considered  part  of  the  offeror’s  proposal  and  subject  to  evaluation  by  the  Government. 
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CCPDS-R  SOFTWARE  ENGINEERING  EXERCISE  (SEE)  SPECIFICATION 
1.0  SCOPE 

The  exercise  system  will  create  scenarios  under  user  direction  and  will  simulate  the  CCPDS-R 
missile  warning  (MW)  capability  in  real  time. 


2.0  APPLICABLE  DOCUMENT 

CCPDS-R  Sysiem  Specincaiion,  section  3.3,  dated  30  September  1986. 


3.0  REQUIREMENTS 


3.1  GENERAL  DESCRIPTION 

The  exercise  system  shall  maintain  MW  information  and  display  the  information  in  tabular  form 
in  real  time.  Specifically,  the  exercise  system  shall  create  scenarios  under  user  direction  and  store  each 
created  scenario  in  a  separate  scenario  file.  It  shall  use  a  generated  scenario  to  nm  the  MW  simulation  in 
real  time.  The  system  shall  provide  the  capability  for  the  user  to  run  a  simulation  while  editing, 
deleting,  creatin'  '"'mg,  or  querying  a  scenario  file  (possibly  the  same  file).  The  design  for  the 
exercise  system  shall  be  modular  to  facilitate  changes  in  software  components  which  are  needed  to 
accommodate  future  changes  in  operational  requirements. 


3.2  HARDWARE 

The  exercise  system  will  generate  tabular  displays  only.  No  special  graphics  hardware  or 
capabilities  shall  be  used.  The  user  interface  shall  be  designed  to  operate  on  a  single  dumb  terminal 
with  keyboard  entry  device. 


3.3  SIMULATION  DATA 


3.3.1  TW/AA  Configuration 

The  TW/AA  configuration  to  be  simulated  shall  be  as  follows: 

1 .  There  shall  be  one  command  center,  designated  as  NORAD. 

2 .  There  shall  be  seven  sensors,  designated  as  PAVE  PAWS  EAST  (PPE),  PAVE 
PAWS  WEST  (PPW),  BMEWS  I,  BMEWS  II,  BMEWS  III,  IR  I,  and  IR  II. 
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3 .  There  shall  be  hve  missile  launch  origin  locations,  designated  as 

MLOCl  through  MLOCS,  and  five  predicted  impactAiuclear  detonation  locations, 
designated  as  IPLOCl  through  IPLOC5. 

4 .  Sensor  connectivity  shall  be  from  each  sensor  to  the  command  center. 

5 .  The  exercise  system  shall  simulate  the  transmission  and  processing  delay  incurred 
from  the  time  a  sensor  transmits  a  missile  warning  message  until  the  message  has  been 
processed  by  the  system  and  made  ready  for  display.  The  processing  delay  parameter  shall 
be  user  selectable  from  0-99  seconds  and  shall  te  constant  during  a  given  missile  warning 
simulation. 

3.3.2  Missile  Warning  Data 

Missile  warning  data  shall  consist  of  missile  launches  and  nuclear  detonations  (NUDETs).  A 
missile  launch  message  shall  consist  of  launch  origin  location,  launch  type  (ICBM,  SLBM),  reporting 
sensor,  position  of  predicted  impact,  and  time  of  launch.  Each  launch  shall  be  detected  by  (i.e., 
associated  with)  only  one  sensor.  A  nuclear  detonation  message  shall  consist  of  time  and  location. 
Launch  locations  and  impact  locations  shall  be  designated  as  described  in  3.3.1. 


3.4  DISPLAY  FORMATS 

Display  formats  shall  consist  of  menus  for  the  user  interface  and  tabular  displays. 

3.4.1  User  Interface 

The  user  interface  shall  be  menu  driven  and  user  friendly.  All  user  inputs  shall  be  validated  for 
proper  format  and  range  of  values.  The  user  shall  be  tK)tified  of  any  entries  that  are  erroneous  or  that 
cannot  be  processed  for  any  other  reason.  Error  messages  shall  be  self-explanatory  and  shall  specify, 
to  the  extent  practical,  the  cause  and  location  of  the  error. 

General  user  capabilities  to  be  provided  shall  include  the  capability  to  start  and  stop  a  session; 
the  capability  to  terminate  the  scenario  generator  (SG)  and/or  missile  warning  simulator  (MWS)  and  exit 
to  the  main  menu  upon  user  request;  the  capability  to  display  the  directory  of  scenario  file  names;  the 
capability  to  select  the  processing  delay  parameter  (see  3.3.1);  and  the  capability  to  interface  with  the 
scenario  generator  and  miss”.w  warning  simulator  as  described  in  3.5  and  3.6,  respectively. 

All  user  inputs  shall  be  ackrxrwledged  within  one  second  of  the  input  For  data  entered  by  the 
user,  the  time  from  completion  of  entry  until  the  database  is  modified  to  reflea  the  update  shall  not 
exceed  two  seconds.  An  advisory  shall  be  provided  within  two  arxl  one  half  seconds  if  the  system 
carmot  complete  such  an  update.  At  a  minimum,  these  performance  requirements  shall  be  met  on 
dedicated  processing  equipment  and  with  at  least  twenty  stored  scenario  files,  consisting  on  the  average 
of  5,000  combined  missile  launch  and  NUDET  events. 
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3.4.2  Tabular  Displays 

The  exefcise  system  shall  be  able  to  generate  three  displays  for  MW  data:  a  missile  launch 
summary  display,  a  predicted  impact/NUC£T  summary  disfday,  and  a  message  display.  The  summary 
di^lays  shall  present  the  MW  information  received  by  the  command  center  as  goierated  by  a  selected 
scenario,  sununarized  from  the  start  of  the  scenario,  in  real  time,  and  in  accordance  with  the  specified 
processing  delay  (see  3.3.1).  The  formats  for  the  missile  launch  summary  display  and  the  predicted 
impact/NUDET  summary  display  shall  be  as  specified  in  figures  1  and  2,  resp^vely.  The  message 
display  shall  sequentially  list  the  messages  received  by  the  command  center,  as  received  in  real  time. 
The  capability  shall  be  provided  to  display  the  corttents  of  at  least  the  five  most  recently  received 
messages  in  foe  scenario.  EMsplay  updates  shall  be  processed  and  reflect  a  scenario  event  within  one- 
half  second  of  the  activation  time  of  the  evaiL  (Activation  time  is  defined  in  section  3.5.) 


3.5  SCENARIO  GENERATOR 

The  SG  shall  only  be  activated  and  deactivated  as  a  result  of  user  actioa  The  SG  shall  be  able 
to  create,  delete,  edit  and  save  files  containing  scenario  data.  Edit  capabilities  for  a  selected  scoiario  file 
shall  include  changing  the  contents  of  events  in  foe  scenario  file,  adding  events  to  foe  scenario  file,  and 
deleting  events  from  the  scenario  file.  The  capability  shall  be  provided  to  save  a  scenario  and  any 
changes  to  it  as  a  new  file  or  as  the  current  file.  Each  event  in  a  scenario  shall  have  a  unique  activation 
time  to  foe  nearest  tenth  of  a  second,  where  foe  activation  time  represents  the  time  foe  reporting  sensor 
transmits  foe  missile  warning  message.  The  user  shall  be  precluded  from  entering  multiple  events  into  a 
scenario  with  foe  same  aaivation  time.  The  user  shall  be  able  to  query  an  individual  scenario  file  to 
search  for  events  based  on  reporting  sensor  and/or  time  of  event  activation.  The  design  for  the  exercise 
system  shall  be  flexible  to  allow  as  future  growth  foe  capability  to  perform  this  query  across  all  scenario 
files.  The  SG  shall  accept  inputs  from  foe  keyboard  to  perform  the  above  fimctions.  There  shall  be  a 
default  scenario  file  consisting  of  a  total  of  5,000  individual  missile  lautKh  and  NUDET  events  and  their 
associated  times  of  aaivation  covering  a  twenty  minute  scenario  period.  The  SG  shall  support  a  total  of 
at  least  40,000  missile  launch  events  and  10,000  NUDET  events  contained  in  one  or  more  scenarios. 


3.6  MW  SIMULATION 

The  MWS  shall  provide  the  user  with  the  capability  to  selea  and  run  a  scenario  contained  in  a 
scenario  file.  The  MWS  shall  nm  this  scenario  in  re^  time,  generating  the  missile  launch  summary 
display,  foe  predicted  impact/NUDET  summary  display,  or  the  message  display,  as  specified  by  the 
user.  The  MWS  shall  be  activated  or  deactivate  oiily  upon  user  request  C^bilities  shall  be  provided 
for  foe  user  to  select  foe  processing  delay  parameter  (see  3.3.1),  to  suspend  the  simulation,  to  resume 
the  simulation,  to  fast  forward  the  simulation  (where  fast  forward  means  the  run  time  between  event 
aaivations  is  reduced  by  two),  and  to  stop  the  fast  forward  capability  and  return  to  foe  normal  run  time 
between  event  aaivations.  The  user  shall  also  have  the  capability  to  select  which  of  the  three  MW 
displays  he  wishes  to  view,  and  to  move  to  other  displays  while  the  simulation  is  running. 
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NUMBER  OF  MISSILE  LAUNCHES 

SENSOR  MISSILE  LAUNCH  ORIGIN 

MLOCl  MIjOC2  MLOC3  MLOC4  MLOC5 


PPE 

PPW 

BMEWSI 
BMEWSn 
BMEWS  ni 
IRI 
IRII 


TOTAL 


FIGURE  I.  MISSILE  LAUNCH  SUMMARY 


IMPACT/NUDET  LOCATIONS 

IPLOCl  IPLOC2  IPLOC3  IPLOC4  IPLOC5 

PREDICTED 
IMPACTS  (PI) 

PPE 

PPW 

BMEWSI 
BMEWS  n 
BMEWS  ni 
IRI 
IR II 


HGURE  2.  PREDICTED  IMPACT/NUDET  SUMMARY 
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3.7  SIMULTANEOUS  GENERATION  AND  SIMULATION 

The  exercise  system  shall  provide  the  capability  for  the  user  to  run  the  MWS  and  SG 
simultaneously,  either  on  the  same  or  different  scenario  files,  while  still  meeting  the  performance 
requirements  specified  herein.  Formats  for  the  di^lays  when  both  are  nmning  simultaneously  will  be 
contractor  defined  as  part  of  the  design  effort 

When  both  the  SG  and  the  MWS  are  processing  the  same  scenario,  the  MWS  displays  shall 
reflect  a  modification  to  an  everu  in  the  scenario  only  if  the  event  has  not  yet  been  processed  by  the 
MWS;  otherwise,  the  MWS  displays  shall  not  reflect  the  changes. 
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CCPDS-R  IFPP  SEE  MATERIAL 


This  appendix  contains  the  information  included  in  the  CCPDS-R  RFP  Instructions  for 
Proposal  Preparation  for  incorporating  the  software  engineering  exercise  as  part  of  the  CCPDS-R 
source  selection.  This  information,  provided  to  the  offerors  in  the  initial  release  of  the  RFP,  identiiies 
the  requirement  for  all  offerors  to  carry  out  the  SEE  as  part  of  the  CCPDS-R  proposal  effort  It  also 
provides  a  high-level  set  of  instructions  detailing  to  the  offerors  what  is  expected  of  them  in  carrying  out 
the  SEE. 
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Software  Engineering  Exercise  (SEE).  The  offeror  shall  carry  out  a  software  engineering 
exercise  which  will  be  defined  by  the  Government  [The  attachment]  contains  the  general  ground  rules 
for  the  conduct  of  the  SEE  and  a  brief  description  of  the  SEE  products  to  be  generated  and  submitted  by 
the  offeror  for  Government  evaluatioa  The  Government  will  provide  die  SEE  specification  and  the 
detailed  SEE  ground  rules  following  receipt  of  proposal,  at  which  time  the  SEE  shall  commence. 
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SOFTWARE  ENGINEERING  EXERCISE 
Preliminary  Instructions  for  the  Offeror 


1.0  PURPOSE 

The  purpose  of  the  software  engineering  exercise  (SEE)  is  to  permit  the  Government  to  evaluate 
an  actual  application  of  each  offeror’s  software  development  methodology  as  proposed  for  the 
CCPDS-R  full-scale  development/production  (FSD/P)  phase.  The  SEE  will  concentrate  exclusively  on 
the  offerors'  approach  to  requirements  analysis,  design,  and  their  interrelationship.  The  offeror’s 
approach  to  implementation,  integration,  test,  quality  assurance,  configuration  management,  and  other 
development  activities  not  explicitly  mentioned  in  the  following  paragraphs  will  not  be  evaluated  by  the 
Government  as  part  of  the  SEE. 


2.0  INSTRUCTIONS  FOR  THE  OFFERORS 

Each  offeror  will  provide  a  prototypical  example  of  his  proposed  software  development 
approach,  as  applied  to  a  sample  problem  t^en  from  the  missile  warning  domain.  The  Government 
will  define  the  sample  problem  and  provide  the  SEE  problem  specification  to  the  Offeror  following 
receipt  of  proposal.  In  performing  the  exercise,  the  offeror  shill  comply  with  aU  provisions  of  his 
proposed  Software  Development  Plan  and  with  section  3.3  of  the  CCPDS-R  System  Specification; 
deviations  shall  be  noted  by  the  offerors. 

Participation  in  the  exercise  shall  be  limited  to  those  individuals  identified  in  the  offeror’s 
proposal  as  part  of  the  CCPDS-R  full-scale  development  team.  Subcontractors  who  will  be  responsible 
for  software  development  on  CCPDS-R  shall  be  active  participants.  Consultants  shall  be  precluded 
from  participating. 

Each  offeror  will  be  allocated  a  period  of  four  (4)  calendar  weeks  from  receipt  of  the  exercise 
materials  until  delivery  to  the  Government  of  all  requested  materials  in  the  formats  described  below. 

The  Government  will  review  this  material  for  a  period  of  time  not  to  exceed  two  (2)  calendar  weeks. 
Following  completion  of  Government  review,  the  Government  will  conduct  an  on-site  visit  at  the 
offeror’s  facility,  at  which  time  the  offeror  shall  brief  his  methodology  approach  to  the  Government  and 
provide  responses  to  Government  requests  for  clarification.  The  Government  will  coordinate  the 
schedule  for  the  on-site  visit  with  the  offeror  upon  receipt  of  the  offeror’s  exercise  results.  Note  that 
there  will  be  no  interaction  between  the  offeror  and  the  Government  during  the  four  week  exercise 
period.  Should  the  offeror  have  any  questions  on  the  exercise,  the  offeror  is  instructed  to  identify 
appropriate  assumptions,  to  note  these  assumptiems,  and  proceed  vrith  the  exercise  based  on  those 
as.sumptions. 
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3.0  PRODUCTS  OF  THE  EXERCISE 

At  the  conclusion  of  the  exercise,  the  offeror  shall  deliver  the  following  items  to  the 
Government: 

a.  A  complete  software  architecture  for  the  sample  problem 

b.  For  two  or  more  offeror-selected  components  of  the  system,  all  requirements  analysis 
conclusions  reached  and  documentation 

c.  For  two  or  more  offeror-selected  components  of  the  system,  all  preliminary  design 
documentation,  including  requirements  traceability,  Ada-based  design  language  (ADL) 
listings,  and  graphics  products 

d .  For  at  least  one  offeror-selected  componait  of  the  system,  all  detailed  design 
documentation,  including  requirements  traceability,  ADL  listings,  and  graphics  products. 

All  textual  products  of  the  exercise,  including  requirements  ^alysis  conclusions  and 
documentation,  ADL  listings,  and  other  design  documentation  shall  be  delivered  to  the  Government 
both  in  hardcopy  form  and  in  machine-readable,  9-track  1600/6250  l^i  tape  format  in  accordance  with 
ANSI  X3.27-1978.  Exception  will  be  made  for  materials  which  the  offeror  does  not  propose  to  create 
and/or  maintain  online  during  the  CCPDS-R  FSD/P  contract  In  particular,  graphical  representations 
shall  be  submitted  in  hardcopy  form.  The  offeror  shall  provide  six  (6)  copies  of  all  hardcopy  products. 
The  products  delivered  shall  be  clear,  coherent,  legible,  and  prepared  in  sufficient  detail  for  effective 
evaluation.  Elaborate  documentation,  expensive  binding,  detailed  art  work  or  other  embellishments  are 
unnecessary. 

In  addition  to  the  delivered  products  described  above,  the  offeror  shall  provide  a  briefing  to  the 
Government  that  summarizes  his  experience  in  the  carrying  out  the  exercise  and  describes  the  products 
produced.  The  briefing  shall  not  exceed  three  (3)  hours  in  duration.  The  topics  presented  shall  include 
the  following: 

I.  Management  approach 

II.  An  overview  of  the  requirements  analysis  approach 

III.  An  overview  of  the  approach  to  preliminary  atxi  detailed  design 

IV.  Other  topics  to  be  determined  by  the  offeror 

The  briefing  to  the  Government  shall  be  presented  between  one  and  two  calendar  weeks  after 
delivery  to  the  Government  of  the  products  of  the  exercise  described  in  points  (a)  -  (d)  above.  The 
briefing  shall  ixit  include  any  discussion  of  further  work  which  the  offeror  may  have  completed 
following  completion  of  the  four  week  SEE  period.  All  participants  in  the  exercise  shall  be  present  at 
the  briefing  to  respond  to  Government  requests  for  clarification.  All  offeror  responses  to  these 
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Government  clarification  requests  together  widi  the  Mefing  presentation  material  and  the  products 
identified  in  items  (a)-(d)  above  shall  be  considered  part  of  ^  offeroi's  proposal  and  subjea  to 
evaluation  by  the  Government 

4.0  SCOPE  OF  THE  SEE 

The  Government  will  not  evaluate  the  following  items: 

a.  Additional  work  accom[dished  on  the  SEE  after  the  initial  4-week  period 

b.  Level  of  staffing 

c.  Measures  of  productivity  and  collection  of  software  development  metrics 

d .  Issues  that  relate  to  coding,  integratitm,  and  test 
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APPENDIX  C 


CCPDS-R  SECTION  M  SEE  MATERIAL 


Listed  below  is  the  material  included  in  the  CCPDS-R  RFP  section  M,  evaluaticHi  criteria,  for 
the  SEE.  This  material  identifies  on  what  basis  an  offeror's  SEE  products  will  be  judged  by  the 
Government. 


Item:  Software  Engineering  Exercise 

The  offeror  will  be  evaluated  on  his  familiarity  with  the  selected  software 
development  methodology  aixl  on  his  capalnlity  to  utilize  Ada.  The  offeror  will  be 
evaluated  on  his  corporate  Ada/Software  Engineering  expertise;  his  requirements 
analysis  and  design  approaches  and  their  inter-relationships;  the  robustness  and 
cohesion  of  his  requirements  analysis  and  design  methodologies;  his  familiarity 
and  expertise  with  the  methodologies;  his  familiarity  with  the  tool  set  and  the 
development  environment;  the  robustness,  cohesion,  and  completeness  of  his 
exercise  design;  his  ability  to  address  and  analyze  real-time  requirements  and 
issues;  his  clarity  and  communication  of  design,  including  the  use  of  ADL  to 
express  design;  and  his  compliance  with  the  exercise  specification  requirements 
and  SDP.  A  visit  to  each  offeror  will  be  scheduled  approximately  six  (6)  weeks 
after  receipt  of  proposals  to  evaluate  the  software  engineering  exercise.  The 
evaluation  will  be  considered  as  pass/fail;  there  will  be  no  opportunity  to  re¬ 
accomplish  the  exercise.  The  visiting  Government  team  will  be  assisted  by 
personnel  from  MITRE  and  the  Software  Engineering  Institute. 


It  should  be  noted  that  while  the  Software  Engineering  Institute  (SEI)  was  identified  in  section  M  as  a 
possible  member  of  the  Government  SEE  evaluation  team,  rx)  representatives  of  the  SEI  did  in  fact 
participate. 
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APPENDIX  D 


SEE  QUESTIONNAIRE 


This  appendix  contains  the  optional  questioimaiie  which  was  submitted  to  all  CCPDS-R  FSD/P 
SEE  offerors. 
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SEE  QUESTIONNAIRE 


THE  PURPOSE  OF  THIS  QUESTIONNAIRE  IS  TO  ASSESS  THE  BENEFITS  OF  THE  SEE 
FROM  THE  OFFERORS’  PERSPECTIVE  AND  TO  EVALUATE  ITS  POTENTIAL  BENEFITS  ON 
FUTURE  ACQUISITIONS. 

PART  I.  PLEASE  CIRCLE  YOUR  RESPONSE  FOR  EACH  OF  THE  QUESTIONS  BELOW. 

1 .  Overall,  the  SEE  was 

a.  beneficial 

b.  somewhat  beneficial 

c.  not  beneficial 

2 .  The  instructions  to  the  offeror  were 

a.  adequate 

b.  mai^al 

c.  not  adequate 

3 .  The  SEE  system  specification  was 

a.  ad^uate 

b.  mai^al 

c.  not  adequate 

4 .  The  scope  of  the  SEE  was 

a.  too  broad 

b.  satisfactory 

c.  too  nareow 

5.  The  SEE,  as  a  technical  problem  to  be  solved,  was 

a.  overly  challenging 

b.  adequately  challenging 

c.  trivid 

6 .  The  SEE,  relative  to  the  CCPDS-R  acquisition,  was 

a.  relevant 

b.  somewhat  relevant 

c.  not  relevant 

7 .  The  time  allotted  for  the  SEE  was 

a.  too  long 

b.  adequate 

c.  too  short 

8 .  The  requirement  to  use  subcontractors  on  the  SEE  was 

a.  beneficial 

b.  not  beneficial 

c.  detrimental 
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SEE  QUESTIONNAIRE  (Continued) 


9 .  The  instnictions  given  for  the  briefing  were 

a.  adequate 

b.  somewhat  adequate 

c.  not  adequate 

10.  The  questions  you  were  given  the  day  before  the  briefing  were 

a.  too  numerous 

b.  adequate  in  number 

c.  too  few 

1 1 .  The  questions  you  were  given  the  day  before  the  briefing  were 

a.  relevant 

b.  somewhat  relevant 

c.  not  relevant 

12.  The  questions  you  were  given  during  the  briefing  were 

a.  too  numerous 

b.  adequate  in  number 

c.  too  few 

1 3.  The  questions  you  were  given  during  the  briefing  were 

a.  relevant 

b.  somewhat  relevant 

c.  not  relevant 

14.  Assembling  the  SEE  team  was 

a.  difficult 

b.  scxnewhat  difficult 

c.  not  difficult 

1 5.  Have  members  of  the  SEE  team  worked  together  previously? 

a.  yes 

b.  some  members  have 

c.  no 


PART  II.  DESCRIBE  THE  MAJOR  BENEFITS  YOU  GOT  FROM  PARTICIPATION  IN  THE  SEE. 
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SEE  QUESTIONNAIRE  (Concluded) 


PART  III.  FOR  EACH  OF  THE  FOLLOWING  PHASES  OF  ECVELOPMENT  FOR  THE  SEE. 
DESCRIBE  THE  LEVEL  OF  EFFORT  SPENT  IN  EACH  PHASE  (PERCENT  OF  TOTAL  SEE 
EFFORT)  AND  ANY  DHTICULTIES  YOU  RAN  INTO  DURING  EACH  PHASE.  ALSO, 
IDENTIFY  THE  TOTAL  EFFORT  G-E..  NUMBER  OF  STAFF-MONTHS  EXPENDED  ON  THE 
SEE). 


A.  Requirements  Analysis 

B.  Top-Level  Design 

C.  Eietailed  Design 

D.  Prototyping 

E.  Briefing 


PART  IV.  WAS  THERE  ANYTHING  YOU  WOULD  HAVE  LIKED  THE  GOVERNMENT  TO 
HAVE  SEEN  IN  THE  SEE  PRODUCTS  BUT  THERE  WAS  NO  PLACE  TO  PUT  IT? 


PART  V.  HOW  SHOULD  THE  SEE  BE  MODIFIED  TO  INCREASE  ITS  BENEFITS  TO  FUTURE 
ACQUISITIONS? 


PART  VI.  USE  THIS  SPACE  FOR  ANY  ADDITIONAL  COMMENTS. 
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GLOSSARY 


ADL 

ANMCC 

ANSI 

AFR 

AFSC 

Asai 

CCPDS 

CCPDS-R 

ODlO 

CDR 

CMAFB 

CRISD 

CSC 

CSCI 

C«DD 

EX 

MD 

DOD 

EASE 

ESD 

FCA 

FM 

FSD/P 


Acronyms 


Ada-based  design  language 

Alternate  National  Military  Command  Center 

American  National  Standairls  Institute 

Air  Force  Regulation 

Air  Force  Systems  Commatxl 

American  Standard  Code  for  Information  Interchange 

Commatxl  Center  Processing  and  Display  System 

Command  Center  Processing  and  Display  System-Replacement 

cotx^ept  definition/design 

critical  design  review 

Cheyenne  Mountain  Air  Force  Base 

computer  resources  integrated  support  document 

computer  software  component 

computer  software  coniiguration  item 

database  design  document 

data  item 

data  item  description 
Department  of  Defense 
ESD  acquisition  support  environment 
Electronic  Systems  EMvision 
functional  configuration  audit 
file  manager 

full-scale  development/production 
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GLOSSARY  (Continued) 
Acronyms 


ICBM 

intercontinental  ballistic  missile 

IDD 

interface  descripticm  document 

IFPP 

instructions  for  proposal  preparation 

IRS 

interface  tequircments  speciiicaticm 

rrw&A 

integrated  tactical  warning  and  attack  assessment 

LAN 

local  area  network 

MWS 

missile  warning  simulator 

NMCC 

National  Military  Command  Center 

NORAD 

North  American  Aerospace  Defend  Command 

NUDET 

nuclear  detonation 

OPCC 

Offutt  Processing  and  Correlation  Center 

OOD 

object-oriented  design 

PCA 

physical  configuration  audit 

PDS 

processing  and  display  subsystem 

PDR 

preliminary  design  review 

RFP 

request  for  proposal 

SAC 

Strategic  Air  Command 

sas 

Survivable  Communications  Integration  System 

SCMP 

software  cottfiguration  management  plan 

SDDD 

software  detailed  design  document 

SDF 

software  development  file 

SDP 

software  development  {dan 

SEE 

software  engineering  exercise 
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GLOSSARY  (Concluded) 
Acronyms 


SEI 

Software  Engineering  Institute 

SG 

scenario  generator 

SLBM 

sea-  or  submarine-launched  ballistic  missile 

SOW 

statement  of  work 

SPS 

software  product  specification 

SQEP 

software  quality  evaluation  plan 

SRR 

system  requirements  review 

SRS 

software  requirements  specification 

SSEB 

source  selection  evaluation  board 

SSPM 

software  standards  and  procedures  manual 

SSR 

software  specification  review 

SSS 

system  segment  specification 

STD 

standard 

STDC 

software  test  descriptions 

STLDD 

software  top-level  design  document 

STP 

software  test  plan 

STPR 

software  test  procedures 

STR 

software  test  report 

TRR 

test  readiness  review 

TW/AA 

tactical  warning  and  attack  assessment 

USI 

user-system  interface 

VDD 

version  description  document 
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